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The  purpose  of  this  thesis  was  to  assess  the  effect  of 
feedback  on  the  utility  of  the  Cost  Performance  Report 
(CPR).  An  experimental  setting  was  used  to  investigate  the 
effect  of  providing  feedback  to  preparers  of  the  CPR. 

Two  separate  experiments  were  conducted  to  collect  data 
concerning  the  effect  of  feedback.  One  experiment  included 
government  personnel  while  the  other  included  contractor 
personnel.  The  results  of  the  research  suggest  that  there 
is  no  statistical  difference  in  the  quality  of  the  CPR 
variance  analyses  when  one  provides  feedback  in  the  form 
given  during  the  experiments. 
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Abstract 

The  purpose  of  this  thesis  was  to  assess  the  effect  of 
feedback  on  the  utility  of  the  Cost  Performance  Report 
(CPR).  The  effort  consisted  of  administering  a  controlled 
experiment  to  two  groups:  a  gathering  of  contractor  per¬ 
sonnel  and  a  group  of  Government  employees.  Both  groups  had 
some  degree  of  knowledge  in  the  area  of  variance  analysis 
reporting . 

Each  experiment  consisted  of  two  randomly  assigned 
instruments,  the  feedback  instrument  (given  to  the  experi¬ 
mental  group)  and  the  non-feedback  instrument  (given  to  the 
control  group). 

The  feedback  used  in  this  experiment  was  similar  in 
structure  to  the  Defense  Acquisition  Executive  Summary  For¬ 
mat  11,  specifically  as  formatted  in  the  Program  Director’s 
Assessment  Review.  Both  of  these  reports  are  used  by 
government  program  offices  to  report  contract  cost  and 
schedule  information  to  higher  levels  of  the  government 
procurement  community. 

The  experiment  required  the  subjects  to  perform  a  task 
similar  to  that  performed  during  the  preparation  of  a  CPR 
and  to  provide  a  response  in  the  form  of  a  variance  analy¬ 
sis.  The  responses  were  rated  by  three  experts  and  the  mean 
values  of  the  feedback  responses  and  non-feedback  responses 
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were  tested  to  determine  if  a  statistically  significant  dif¬ 
ference  existed  between  the  two  means. 

No  statistically  significant  difference  was  found  to 
exist  between  the  two  means  suggesting  that  providing  feed¬ 
back  to  subjects,  as  given  in  this  experiment,  has  no  effect 
on  the  quality  of  the  variance  analyses  they  prepare. 

Although  supporting  the  fact  that  feedback  as  a  whole 
is  beneficial,  the  results  of  this  thesis  suggest  that  in 
order  to  produce  a  statistically  significant  difference  in 
the  quality  of  variance  analyses,  the  feedback  given  in 
these  experiments  needs  to  be  reevaluated. 

A  recommendation  was  made  to  conduct  a  follow-on  thesis 
with  an  improved  feedback  measure  possibly  involving  a 
feedback  briefing,  an  increase  in  time  allotted  to  the 
experiment,  or  a  change  to  the  feedback  vocabulary. 
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THE  EFFECT  OF  FEEDBACK  ON 


COST  PERFORMANCE  REPORT  UTILITY 

I.  Introduction 


General  Issue 

The  Cost  Performance  Report  (CPR)  is  currently  the  main 
vehicle  by  which  the  U.S.  Government  collects  cost  and 
schedule  information  on  major  defense  contracts.  Guidelines 
for  preparation  of  this  report  are  outlined  in  Department  of 
Defense  Data  Item  Description,  DI-F-6000C,  which  is  readily 
available  to  defense  contractors.  However,  much  of  the 
literature  in  CPR  utility,  addressed  in  Chapter  II  of  this 
proposal,  suggests  that  a  high  percentage  of  defense  con¬ 
tractors  fail  to  provide  adequate  background  information  in 
support  of  cost  and  schedule  performance  as  required  by  the 
guidelines.  Since  the  CPR,  if  properly  prepared,  provides 
useful  information  to  high-level  decision  makers,  it  is 
imperative  to  improve  its  value  to  the  decision-making 
process . 

Problem  Statement  &  Hypothesis 

Although  defense  contractors  are  provided  with  the 
instructions  for  the  preparation  of  Cost  Performance  Re¬ 
ports,  the  usefulness  of  the  information  contained  in  these 
reports  is  limited.  A  review  of  the  literature  in  Chapter 
II  will  identify  a  number  of  sources  who  claim  that  the  lack 
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of  CPR  utility  is  not  a  result  of  providtng  improper  in¬ 
structions  to  the  contractors,  but  instead  due  to  a  lack  of 
appreciation,  on  the  part  of  a  number  of  defense  contrac¬ 
tors,  of  the  importance  of  the  information  they  include  in 
their  submissions  to  the  Government. 

The  second  part  of  the  literature  review  addresses  the 
effect  of  feedback  on  the  quality  of  performance  reports. 

As  detailed  further  in  Chapter  II,  it  appears  that  providing 
feedback  to  subjects  on  the  usefulness  of  performance  re¬ 
ports  prior  to  their  preparation  by  the  subjects  results  in 
a  higher  degree  of  accuracy  in  the  performance  reports.  The 
author  hypothesizes  that  by  applying  this  information  in  the 
area  of  CPRs,  the  overall  quality  of  the  CPRs  will  improve. 

Specifically,  the  effect  that  feedback  has  on  the 
quality  of  the  CPRs  submitted  to  the  Government  needs  to  be 
adequately  measured.  At  the  core  of  this  thesis  is  an 
experiment  to  study  if  there  is,  in  fact,  a  positive  effect 
in  having  the  Government  provide  feedback  to  contractor 
personnel  who  prepare  the  CPR.  The  author’s  hypothesis  is 
that  providing  feedback  to  these  individuals  will  improve 
the  quality  of  the  variance  analyses  contained  in  the  CPRs. 
If  a  positive  effect  does  exist,  this  information  should  be 
applied  throughout  the  acquisition  environment. 

Investigative  Questions 

The  following  questions  are  relevant  in  addressing  the 
effect  of  feedback  on  CPR  utility: 
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1)  What  specific  background  information  do  high-level 
decision-makers  use  in  making  their  decisions? 

2)  How  should  this  information  be  formatted  to  include 
it  as  feedback  to  preparers  of  CPRs? 

3)  Who  should  serve  as  subjects  for  this  experiment? 

4)  How  does  feedback  improve  the  quality  of  perfor¬ 
mance  reports? 

Justification  of  the  Research 

Recently,  the  Government  has  had  to  endure  sharp  criti¬ 
cism  for  serious  blunders  in  defense  contract  management, 
particularly  in  the  area  of  cost.  Most  people  have  heard 
the  stories  in  the  news  of  Government  mismanagement  result¬ 
ing  in  exorbitant  prices  for  items  normally  costing  several 
dollars.  Of  course,  this  situation  is  embarrassing  to  the 
Department  of  Defense.  More  importantly,  however,  it 
causes  the  public  to  lose  confidence  in  the  ability  of  the 
Defense  Department  to  manage  the  cost  of  defense  contracts. 

The  mismanagement  referred  to  above  is  certainly  the 
exception  to  the  normal  way  of  doing  business.  However,  the 
attention  it  draws  to  the  acquisition  process  enforces  the 
notion  that  the  public  cares  very  much  about  the  efficient 
cost  management  of  defense  contracts.  As  previously 
mentioned,  the  CPR  is  the  document  designated  by  the  Govern¬ 
ment  as  the  main  vehicle  for  obtaining  this  information  from 
defense  contractors. 
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Scope 


The  CPR  falls  under  the  overall  concept  of  the  Cost/ 
Schedule  Control  Systems  Criteria  (C/SCSC).  In  order  to 
properly  address  the  issue  of  CPR  utility  and  specifically 
the  effect  of  feedback  on  variance  analyses  within  the  CPR, 
it  is  important  to  understand  what  C/SCSC  entails. 

This  thesis  does  not  provide  a  validation  of  C/SCSC  as 
a  whole,  but  instead  accepts  the  ideas  incorporated  in  this 
discipline  as  appropriate  for  collecting  contract  cost  and 
schedule  information.  As  an  integral  part  of  C/SCSC,  the 
CPR  has  been  in  use  for  numerous  years  and  the  author  as¬ 
sumes  that  if  properly  prepared,  the  CPR  provides  the  infor¬ 
mation  government  decision-makers  want,  and  actually  need, 
to  make  the  best  decisions  possible  concerning  major  defense 
acquisitions. 

Background 

During  the  late  1960s,  Government  program  managers  were 
faced  with  the  highly  complex  task  of  managing  large  defense 
contracts  in  an  unfavorable  atmosphere  of  "increased  techni¬ 
cal  complexity  of  weapon  systems,  the  long  lead  times  in 
procuring  them,  the  environment  of  uncertainty  in  the 
defense  business,  and  the  inflationary  spiral  of  the 
economy"  (16:23). 

Since  there  was  no  existing  requirement  to  manage  con¬ 
tracts  in  a  certain  way,  program  managers  controlled  the 
cost  on  their  contracts  to  the  best  of  their  abilities.  As 
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one  would  expect,  some  were  more  successful  than  others. 

More  importantly,  however,  was  the  realization  by  indi¬ 
viduals  in  charge  of  the  acquisition  process  that  the  lack 
of  adequate  and  uniform  contract  cost  control  needed  to  be 
addressed . 

During  this  time,  two  approaches  to  cost  control 
emerged.  The  first  of  these  was  the  Program  Evaluation  and 
Review  Technique  (PERT).  PERT  is  a  network  scheduling 
technique  applied  by  the  Navy  to  its  Polaris  missile 
program.  At  the  heart  of  PERT,  "was  a  work  breakdown  struc¬ 
ture  that  subdivided  the  program  through  successive  levels 
of  detail,  corresponding  to  the  way  in  which  the  work  was  to 
be  performed"  (18:6). 

The  second  approach  was  the  "Earned  Value"  concept  used 
by  the  Air  Force  on  its  Minuteman  missile  program.  This 
method  adopted  tiie  work  breakdown  structure  idea  of  PERT  and 
added  a  set  of  criteria  that  a  contractor’s  management 
system  should  meet.  These  criteria  specify  guidelines  which 
must  be  met,  but  do  not  require  that  a  contractor  use  a 
particular  management  system.  A  further  improvement  was  the 
establishment  of  a  contract  provision  allowing  for  the 
inspection  of  a  contractor’s  facility  to  verify  proper 
implementation  of  the  system  criteria  (9:14). 

In  June  1966,  the  Air  Force  combined  the  strong  points 
of  PERT  and  the  Earned  Value  concept  and  published  the  Cost/ 
Schedule  Planning  and  Control  Specification  (C/SPCS)  (18:7). 
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The  final  step  in  the  evolution  of  C/SCSC  came  in 
December  of  1967,  when  the  Assistant  Secretary  of  Defense 
(Comptroller)  published  Department  of  Defense  Instruction 
7000.2,  Performance  Measurement  for  Selected  Acquisitions, 
"which  formally  implemented  the  Cost/Schedule  Control  Sys¬ 
tems  Criteria  on  a  defense-wide  basis"  (16:2). 
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II.  Literature  Review 

The  literature  review  section  of  this  thesis  consists 
of  three  parts.  The  first  part  is  a  discussion  of  the 
literature  on  the  Cost/Schedule  Control  Systems  Criteria. 
Part  two  of  the  literature  review  addresses  the  issue  of 
feedback.  Of  particular  importance  to  this  effort  is  its 
effect  on  the  quality  of  performance  reports.  Part  three  of 
this  section  outlines  the  implications  that  the  literature 
review  has  on  the  use  of  feedback  to  improve  CPR  utility. 


Cost/Schedule  Control  Systems  Criteria 

Department  of  Defense  Instruction  7000.2  states  the 
main  objective  of  C/SCSC  as  follows: 

To  provide  an  adequate  basis  for  responsible 
decision-making  by  both  contractor  management  and 
DoD  Components,  contractor’s  internal  management 
control  systems  must  provide  data  which  (a)  in¬ 
dicate  work  progress,  (b)  properly  relate  cost, 
schedule  and  technical  accomplishment,  (c)  are 
valid,  timely  and  auditable,  and  (d)  supply  DoD 
managers  with  information  at  a  practicable  level 
of  summarization.  (7:1) 

The  responsibility  of  ensuring  that  the  contractor 
maintain  a  management  control  system  that  meets  the  cri¬ 
teria  conditions  stated  above  rests  with  the  government 
program  manager.  To  help  alleviate  the  burden  on  the  pro¬ 
gram  manager,  the  Departments  of  the  Air  Force,  Army,  Navy, 
the  Defense  Logistics  Agency,  and  the  Defense  Contract  Audit 
Agency  published  the  Cost/Schedule  Control  Systems  Criteria 
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Joint  Implementation  Guide.  This  document  provides  invalu¬ 
able  information  concerning  the  implementation  of  C/SCSC, 
and  categorizes  the  criteria  as  follows:  1)  Organization, 

2)  Planning  and  Budgeting,  3)  Accounting,  4)  Analysis,  and 
5)  Revisions  and  Access  to  Data  (4:3-1  -  3-20). 

C/SCSC  is  required  for  selected  contracts  within  pro¬ 
grams  designated  as  major  system  acquisitions  as  defined  in 
Department  of  Defense  Directive  5000.1,  Ma.ior  System  Ac¬ 
quisitions  (6:1).  The  present  thresholds  within  the  Air 
Force  above  which  C/SCSC  is  required  are  $40  million  for 
Research,  Development,  Testing,  &  Evaluation  (RDT&E)  con¬ 
tracts  and  $160  million  for  Production  contracts  (3:1-2). 

The  Criteria  Concept.  During  the  initial  implementa¬ 
tion  stages  of  C/SCSC,  the  Government  was  faced  with  a  large 
amount  of  opposition  by  defense  contractors.  The  major 
reason  for  their  reluctance  to  apply  C/SCSC  was  due  to  their 
belief  that  C/SCSC  required  them  to  use  a  particular  system 
for  the  management  of  their  contracts. 

It  took  some  time  for  the  Government  to  convince  con¬ 
tractors  that  C/SCSC  is  not  a  system,  but  rather,  "a  set  of 
criteria  designed  to  define  an  adequate  contractor  cost  and 
schedule  management  control  system.  Changes  to  an  existing 
system  are  required  only  to  the  extent  that  it  does  not  meet 
the  criteria”  (1:45).  Proper  application  of  the  criteria 
assures  the  Government  that  a  contractor’s  system  will 
generate  timely  and  reliable  data. 
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It  is  important  to  briefly  point  out  at  this  time  that 
C/SCSC,  in  and  of  itself,  does  not  require  the  preparation 
of  the  CPR.  However,  by  specifying  criteria  that  contrac¬ 
tors  must  follow,  C/SCSC  provides  a  standardized  framework 
for  collecting  the  data  used  to  generate  the  CPR.  (This 
point  will  be  expanded  upon  in  the  section  on  financial 
reporting . ) 

Earned  Value.  At  the  core  of  C/SCSC  is  the  concept  of 
Earned  Value.  It  is  important  to  understand  what  "earned 
value"  is  because  this  concept  provides  the  basis  for  pei — 
formance  measurement  of  defense  contracts.  In  order  to 
understand  earned  value,  it  is  necessary  to  first  introduce 
some  terminology. 

Hemphill  and  Fleming  state  that  the  following  list 

essentially  comprises  the  "earned  value"  concept: 

BCWS  -  Budgeted  Cost  for  Work  Scheduled 
BCWP  -  Budgeted  Cost  for  Work  Performed 
ACWP  -  Actual  Cost  of  Work  Performed 
EAC  -  Estimate  At  Completion 
METC  -  Monthly  Estimate  To  Complete 
BAC  -  Budget  At  Completion 
Cost  and  Schedule  Variances 
Traceability  (12:28) 

Definitive  guidance  for  most  of  the  terms  listed  above 
can  be  found  in  the  C/SCSC  Joint  Implementation  Guide.  This 
guide  provides  direction  to  both  Government  and  contractor 
personnel  regarding  the  purpose  and  implementation  of 
C/SCSC. 

BCWS,  as  defined  by  the  guide,  is  "the  sum  of  the 
budgets  for  all  work  packages,  planning  packages,  etc., 
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scheduled  to  be  accomplished  (including  in-process  work 
packages),  plus  the  amount  of  level-of-effort  and  appor¬ 
tioned  effort  scheduled  to  be  accomplished  within  a  given 
period  of  time"  (4:2-2).  Prior  to  explaining  the  new  terms 
introduced  by  this  definition,  it  is  important  to  note  that 
BCWS  refers  to  scheduled  work. 

The  "work  package"  is  a  concept  which  allows  for  a  more 
uniform  means  of  work  measurement.  It  denotes  a  particular 
subdivision  of  work  from  which  all  performance  measurement 
is  derived.  The  Joint  Implementation  Guide  states  that  a 
work  package  must  have  the  following  characteristics: 

1)  It  represents  units  of  work  at  levels 
where  work  is  to  be  performed. 

2)  It  is  clearly  distinguished  from  all  other 
packages. 

3)  It  is  assignable  to  a  single  organiza¬ 
tional  element. 

4)  It  has  scheduled  start  and  completion 
dates,  and  as  applicable,  interim  milestones,  all 
of  which  are  representative  of  physical  accom- 

pl ishment. 

5)  It  has  a  budget  or  assigned  value  ex¬ 
pressed  in  terms  of  dollars,  man-hours,  or  other 
measurable  units. 

6)  Its  deviation  is  limited  to  a  relatively 
short  span  of  time  or  it  is  subdivided  by  discrete 
value  milestones  to  facilitate  the  objective  meas¬ 
urement  of  work  performed. 

7)  It  is  integrated  with  detailed  engineer¬ 
ing,  manufacturing,  or  other  schedules.  (4:2-3) 

Unlike  a  work  package,  a  planning  package  has  not  been 

fully  defined.  The  definition  of  a  planning  package,  as 

well  as  those  of  "level  of  effort"  and  apportioned  effort, 

both  referring  to  effort  which  is  not  tied  to  specific  units 

of  work,  is  included  in  the  Joint  Implementation  Guide 
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(4:2-2),  but  further  discussion  of  these  terms  is  beyond  the 
scope  of  this  thesis. 

BCWP  refers  to  "the  sum  of  the  budgets  for  completed 
work  packages  and  completed  portions  of  open  work  packages, 
plus  the  applicable  portion  of  the  budgets  for  level  of 
effort  and  apportioned  effort"  (4:2-2).  Gadeken  and  Tison 
define  BCWP  as  "the  work  actually  accomplished  measured  in 
terms  of  the  budget  planned  for  that  work"  (9:14).  The 
important  thing  to  remember  in  this  case  is  that  BCWP  refers 
to  work  performed  (rather  than  scheduled),  measured  not  by 
the  actual  cost  of  that  work,  but  rather  by  the  amount  origi¬ 
nally  planned  for  that  work  to  cost.  Several  techniques 
exist  for  the  measurement  of  BCWP,  but  discussion  of  them  is 
beyond  the  scope  of  this  thesis. 

ACWP  is  defined  as  "the  costs  actually  incurred  and 
recorded  in  accomplishing  the  work  performed  within  a  given 
time  period"  (4:2-1).  At  the  risk  of  oversimplifying  this 
definition,  it  is  significant  to  point  out  that  ACWP,  refers 
to  work  performed  (rather  than  scheduled);  but  unlike  BCWP, 
it  is  measured  by  the  actual  cost  and  not  the  planned  cost. 

The  Estimate  at  Completion  (EAC)  consists  of  "actual 
direct  costs,  plus  indirect  costs  allowable  to  the  contract, 
plus  the  estimate  of  costs  (direct  and  indirect)  for  author¬ 
ized  work  remaining"  (4:2-2).  The  EAC,  therefore,  is  an 
aggregation  of  actual  costs  to  date  on  completed  work  plus 
the  expected  cost  of  work  to  be  done.  If  one  assumes,  as  is 
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usually  the  case,  that  the  contractor  submits  a  monthly  CPR, 
then  the  second  part  of  the  EAC  definition  above  refers  to 
the  Monthly  Estimate  to  Complete  noted  by  Hemphill  and 
Fleming. 

The  Budget  at  Completion  (BAC)  refers  to  the  overall 
budget  set  aside  for  the  completion  of  a  certain  amount  of 
work.  Specifically,  it  consists  of  "the  sum  of  the  original 
budgets  plus  or  minus  budget  changes  resulting  from  contract 
changes,  internal  replanning,  and  application  of  management 
reserves"  (5:8).  The  contract  BAC  is  a  summation  of  the 
budgets  for  all  the  contract  work  packages  (including  plan¬ 
ning  packages.) 

Until  now  we  have  discussed  only  the  meaning  of  the 
terms  listed  by  Hemphill  and  Fleming.  But  in  order  for 
these  terms  to  be  of  any  value,  they  need  to  be  compared  to 
one  another.  The  resulting  differences  from  comparing  BCWS, 
BCWP,  and  ACWP  are  called  "variances." 

The  term  "variance"  which  is  central  to  the  concept  of 
earned  value,  simply  denotes  a  deviation  from  either  the 
cost  plan  or  schedule  plan  outlined  by  the  contractor.  Both 
cost  and  schedule  variances  discussed  in  the  CPR  are  derived 
from  the  concept  of  earned  value. 

Specifically,  subtracting  BCWS  from  BCWP  results  in  a 
schedule  variance.  A  positive  schedule  variance  indicates  a 
favorable  variance  and  a  negative  variance  indicates  an 
unfavorable  variance  (5:6-7).  Positive  schedule  variances 


12 


result  when  one  is  further  along  in  the  completion  of  a  unit 
of  work  than  one  planned  to  be  at  the  present  (this  does  not 
imply  time,  rather  it  is  a  work  variance).  Schedule  varian¬ 
ces  say  nothing  about  the  actual  costs  incurred. 

Cost  variances  result  from  subtracting  ACWP  from  BCWP. 

A  positive  figure  indicates  a  favorable  variance  and  a  nega¬ 
tive  variance  indicates  an  unfavorable  variance  (5:7).  A 
positive  cost  variance  simply  means  that  the  amount  of  money 
spent  in  completing  a  unit  of  work  is  less  than  the  amount 
budgeted  for  that  work.  Cost  variances  say  nothing  about 
how  far  ahead  or  behind  schedule  one  is. 

The  final  item  listed  by  Hemphill  and  Fleming  is 
"traceability."  This  term  simply  means  that  all  work  needs 
to  be  traceable  to  a  single  responsible  individual.  The 
work  package  concept  addresses  this  requirement  by  stipu¬ 
lating  that  work  packages  must  be  assigned  so  that  one 
individual  manager  is  held  responsible  for  the  work  (4:2-3). 
Traceability  results  in  a  system  of  accountability  for  the 
contractor,  and  an  effective  form  of  visibility  for  the 
Government.  Whenever  a  problem  arises,  both  the  contractor 
and  the  Government  know  where  to  start  looking  for  a  solu¬ 
tion  to  the  problem. 

During  the  early  years  of  C/SCSC  usage,  many  of  its 
proponents  understood  that  contractors  would  resist  the 
introduction  of  new  terms  such  as  the  ones  previously 
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listed.  J.  Stanley  Baumgartner,  Director  of  C/SCSC  Manage¬ 
ment  Courses  for  the  Defense  Systems  Management  College, 
wrote  in  1974: 

If  terms  like  BCWP,  BCWS  and  ACWP  sound  like 
jargon  that  will  soon  go  out  of  style,  two  points 
are  worth  noting.  One  is  that  they  have  been  in 
widespread  use  for  some  years  and  are  destined  to 
be  in  use  for  a  long  time  because  of  general  ac¬ 
ceptance  of  the  criteria  approach.  The  other  is 
that  the  terms  that  are  used  and  their  meaning  is 
well  understood  by  decision  makers  at  high  levels 
of  DOD.  (2:34) 

With  the  possible  exceptions  of  the  BCWP,  and  hence  the 
cost  and  schedule  variances,  the  other  terms  were  not  new  to 
contractors  when  C/SCSC  was  introduced.  Generally,  informa¬ 
tion  regarding  the  incremental  and  cumulative  budgets,  the 
time-phased  plan  for  accomplishing  the  work,  and  the  actual 
costs  of  completed  work  is  readily  available  in  any  good 
management  control  system. 

C/SCSC  and  Financial  Reporting.  As  stated  by  Gadeken 
and  Tison,  the  criteria: 

Do  not  require  the  submission  of  any  reports 
to  the  Government,  but  specify  the  reporting  capa¬ 
bilities  contractors’  internal  systems  must  have 
and  the  types  of  data  that  the  systems  should  be 
able  to  produce.  The  type  and  detail  of  reports 
are  then  selected  by  the  Government  program  office 
after  considering  individual  service  and  program 
requirements.  (9:15) 

The  DoD  currently  requires  the  submission  of  the  Cost 
Performance  Report  for  all  contracts  which  qualify  as  "major 
system  acquisitions"  as  described  earlier.  The  CPR  consists 
of  five  parts,  or  "formats,"  which  incorporate  the  necessary 
cost  and  schedule  information. 
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Format  1  categorizes  the  contract  cost  and  schedule 
data  by  summary  level  work  breakdown  structure  elements. 
Format  2  categorizes  the  data  by  functional  cost  categories. 
Format  3  delineates  the  budget  plan  used  to  measure  the 
contract  performance.  Format  4  addresses  the  manpower 
requirements  to  meet  the  plan.  Finally,  Format  5  provides  a 
narrative  report  which,  among  other  things,  is  used  to 
explain  cost  and  schedule  variances  which  exceed  specified 
thresholds.  Specific  guidance  for  the  preparation  of  this 
report  is  contained  in  DOD  Data  Item  Description  DI-F-6000C 
(5:1-18). 

Effectiveness.  The  criteria  approach  to  C/SCSC  is 
effective  because  it  "offers  flexibility  to  contractors  in 
selecting  systems  and  techniques  best  suited  to  their  own 
internal  needs  while  still  satisfying  the  requirements  of 
the  Government"  (23:38).  Contractors  are  free  to  use  their 
innovation  in  implementing  systems  for  cost  and  schedule 
management  which  do  not  conflict  with  their  internal  pro¬ 
cedures,  yet  provide  the  necessary  information  to  prepare 
the  required  reports  for  the  Government. 

Additionally,  the  ability  of  contractors  to  choose  a 
particular  system  allows  them  the  opportunity  to  minimize 
the  amount  of  redundancy  involved  in  preparing  both  internal 
and  Government  reports.  The  effectiveness  of  C/SCSC  is 
clearly  stated  by  Weisberg: 

Variances  resulting  from  these  comparisons 
provide  DOD  and  contractor  managers  with  cost  and 
schedule  performance  indicators  enabling  them  to: 
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1)  Determine  program  progress  by  specific 
element  of  work. 

2)  Identify  problem  areas  and  their  signifi¬ 
cance  to  the  source  so  corrective  action  can  be 
taken . 

3)  Evaluate  organizational  performance. 

4)  Have  a  factual  basis  for  projecting  future 
cost  and  schedule  performance.  (23:37) 

Problems.  Until  now,  only  the  positive  aspects  of 
C/SCSC  have  been  discussed.  As  may  be  expected,  some  prob¬ 
lems  with  the  criteria  approach  have  been  noted.  Despite 
the  general  acceptance  of  C/SCSC,  there  are  still  some 
problems  concerning  the  criteria  approach.  There  is  still 
"a  lack  of  understanding  in  industry  and  the  Government  of 
C/SCSC  and  how  it  works....  Another  problem  is  the  effort 
required  to  understand  the  terminology"  (2:35). 

In  addition  to  the  problems  pertaining  to  the  criteria 
themselves,  two  major  problems  still  exist  concerning  the 
implementation  of  C/SCSC.  One  concerns  the  C/SCSC  review 
process.  Specifically,  numerous  sources  claim  that  team 
members  are  not  well-qualified  to  perform  contractor  site 
visitations  ( 9 : 1 7 ; 1 6 : 1 73 ; 22 : 1 7  ) . 

The  other  major  problem  area,  and  the  one  of  signifi¬ 
cance  to  this  effort,  was  documented  in  a  1975  study  of 
"corporate  memory"  for  the  years  1968  through  1975.  This 
problem  area  addresses  the  inability  of  both  the  Government 
and  contractors  to  effectively  use  the  CPRs  generated  by 
validated  systems  (16:170).  Webster,  in  his  1987  study, 
reconfirmed  this  area  as  one  requiring  improvement  (22:21). 
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Capt  J.  B.  Holeman  Jr.,  USA,  an  instructor  in  the  Com¬ 
puter  Systems  &  Simulation  Department  of  the  Defense  Systems 
Management  College,  devoted  an  entire  article  to  the  proper 
analysis  of  CPRs.  He  conducted  a  three-month  study  in  which 
he  investigated  the  methods  of  improving  the  use  of  data 
generated  by  defense  contractors  and  reported  in  the  CPRs 
( 13:39-42) . 


Holeman  summarized  the  four  problem  areas  he  noticed 
and  suggested  a  plan  for  improving  the  overall  analysis  of 
these  reports.  The  four  problem  areas  were: 

1.  General  lack  of  effective  communication 
regarding  C/SCSC  analysis  within  the  program  man¬ 
agement  community. 

2.  Reluctance  to  use  the  computer  and/or 
operations  research  approaches. 

3.  Failure  to  understand  and  incorporate 
significant  factors  such  as  overhead,  the  contrac¬ 
tor’s  management  philosophy,  general  economic 
trends  and  the  impact  of  technical  performance  in 
the  total  analysis  picture. 

4.  Difficulty  in  analyzing  schedule  variance 
and  predicting  a  Government  estimated  cost  to  com¬ 
plete,  including  estimated  overrun  or  underrun  for 
all  authorized  work.  (13:39-42) 

Varady  and  turner  offer  a  much  more  critical  view  of  the 
lack  of  contractor  knowledge  concerning  the  CPR: 

It  is  clear  from  involvement  in  implementing 
[C/SCSC]  with  several  contractors  that  they  have 
no  idea  who  gets,  reads  or  grades  the  report. 

Many  contractors  believe  that  the  CPR  submissions 
are  just  piled  on  top  of  the  dozens  (or  hundreds) 
of  other  data  item  reports  that  serve  only  to  keep 
dust  off  an  8-  by  11-inch  piece  of  desk....  These 
reports  have  tremendous  exposure  up  through  the 
highest  levels  of  the  Defense  Department,  and  con¬ 
tractors  must  becjme  aware  that  these  reports  are 
read,  thoroughly  analyzed,  and  briefed  to  people 
who  can  kill  programs  with  the  stroke  of  a  pen. 

(21:12) 
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Dr.  Anthony  Webster,  professor  of  financial  management 
at  the  Defense  Systems  Management  College,  concluded  an 
exhaustive  study  of  C/SCSC  in  1987.  In  addition  to  the 
finding  already  mentioned,  he  found  many  other  areas  where 

f 

C/SCSC  could  be  improved.  Additional  areas  of  interest  to 
this  effort  included  timeliness  of  the  CPRs  and  quality  of 
the  variance  analysis  within  the  CPRs  (22:17).  His  specific 
recommendations  are  discussed  in  the  section  on  the  impli¬ 
cations  of  the  literature. 

Feedback 

This  section  will  begin  with  a  brief  definition  of 
feedback  followed  by  a  discussion  of  the  effect  of  feedback 
in  other  disciplines  and  its  documented  effect  in  the  area 
of  performance  reports. 

Greller  documents  that  literature  on  feedback  falls 
into  the  following  categories:  1)  identifying  the  charac¬ 
teristics  of  effective  feedback;  2)  distribution  of  feed¬ 
back;  and  3)  application  of  feedback  (10:24).  Discussion  on 
the  distribution  of  feedback  is  not  within  the  scope  of  this 
thesis,  but  it  is  beneficial  to  address  the  other  two  cate¬ 
gories  . 

Definition  of  Feedback.  According  to  Huczynski,  much 
of  the  pioneering  work  in  the  area  of  feedback  is  credited 
to  Leon  Festinger  and  his  "social  comparison  theory"  (14:12- 
5).  This  theory  noted  that  individuals  are  very  interested 
in  getting  information  concerning  their  performance.  The 
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name  given  to  this  information  is  "feedback."  Huczynski 
provides  his  own  definition  of  feedback,  calling  it  a  "sho¬ 
rthand  term  for  the  intervention  procedure...  whereby  infor¬ 
mation  about  the  consequences  of  an  individual’s  actions  or 
responses  to  a  task  are  reported  back  to  him  (14:126). 

Tosi ,  providing  a  simpler  definition,  refers  to  feed¬ 
back  as  "the  amount  of  information  that  a  person  gets  about 
the  results  of  a  job"  (20:95).  He  further  states  that 
sources  for  this  information  include  coworkers,  supervisors, 
and  the  job  itself. 

Characteristics  of  Effective  Feedback.  In  order  for 
feedback  to  be  a  valuable  tool,  Tosi  states  that  it  should 
have  certain  characteristics.  Effective  feedback: 

1)  Should  be  as  precise  and  as  specific  as 
possi ble. 

2)  Should  be  timely. 

3)  Should  be  impersonal  rather  than  aimed  at 
personal i ty  traits . 

4)  Should  be  noticeable. 

5)  Should  be  given  frequently.  (20:590-592) 
Application  of  Feedback.  The  application  of  feedback 

and  its  general  effect  on  performance  has  been  documented  in 
various  studies.  Komaki,  Heinzman,  and  Lawson  concluded 
that  feedback  increased  the  level  of  performance  in  a  driver 
safety  program  at  a  city  vehicle  maintenance  facility  (15: 
261-270).  Sul zer-Azaroff  &  de  Santamaria  documented  a 
decrease  in  hazardous  conditions  due  to  the  effective  intro¬ 
duction  of  feedback  at  a  small  factory  specializing  in  the 
manufacturing  of  customized  plastic  parts  (19:518). 
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The  effect  of  feedback  was  also  documented  by  Zemke  and 
Gunkler  in  a  report  which  outlined  the  improvement  in  the 
management  of  a  theme  park  due  to  proper  implementation  of 
feedback  into  the  organization  (24:565-583). 

Feedback  and  Performance  Reports.  Feedback  has  also 
been  applied  in  the  area  of  performance  reports.  All  of  the 
information  in  this  section  is  based  on  an  article  which 
explains  an  experiment  conducted  at  an  air  force  base  in 
Texas.  The  experiment  was  designed  to  determine  the  effect 
of  feedback  on  the  decision-making  behavior  of  mid-level  Air 
Force  officers.  Seventy-five  subjects  were  randomly  assigned 
into  five  experimental  groups.  All  five  groups  were  pre¬ 
sented  with  hypothetical  data  and  asked  to  evaluate  the 
overall  performance  of  the  base  training  wings  during  a 
pre-test.  No  significant  difference  was  found  between  the 
five  groups. 

Prior  to  the  post-test,  groups  two  through  five  re¬ 
ceived  a  policy  statement  which  associated  relative  weights 
to  the  five  goals  of  the  pilot  training  wings.  Additional¬ 
ly,  during  the  post-test,  groups  three  through  five  received 
feedback  indicating  the  decision  reached  by  senior  managers 
on  each  case.  For  group  three,  the  decision  was  consistent 
with  the  policy  statement.  Group  four  received  feedback 
indicating  that  senior  managers  had  ignored  the  policy 
statement.  Random  feedback  was  given  to  group  five. 
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Results  of  this  experiment  concluded  that  mid- level 
managers  exhibited  a  statistically  different  decision-making 
behavior  when  feedback  indicating  senior  manager  evaluations 
was  introduced  (11:833-841). 

Implications  of  the  Literature 

The  C/SCSC  literature  seems  to  suggest  that  the  de¬ 
fense  contracting  community  could  benefit  from  increased 
communication  between  contractors  and  the  Government.  As 
previously  stated,  Webster’s  1987  study  noted  deficiencies 
in  the  timeliness  of  CPRs  and  in  the  quality  of  the  CPR 
variance  analyses.  Among  his  proposed  solutions  in  these 
areas  are  the  following: 

1)  Improvise  ways  to  help  contractor  and 
Government  analyst  to  better  appreciate  value  of 
the  data  and  emphasize  a  greater  understanding  of 
how  to  analyze  and  utilize  the  data. 

2)  Service  focal  points  should  assure  that 
the  contractor  is  aware  of  the  need  for  timely  and 
quality  data  for  the  reports  to  yield  their  maxi¬ 
mum  utility. 

Although  Webster’s  solutions  do  not  specifically  ad¬ 
dress  the  use  of  feedback,  one  can  view  his  second  point  and 
interpret  from  the  information  in  the  C/SCSC  and  feedback 
literature  that  feedback  would  be  appropriate  to  help  cor¬ 
rect  the  deficiencies  he  notes.  Feedback  has  been  found  to 
be  beneficial  in  many  different  situations  and  specifically 
in  the  area  of  performance  reports. 

By  its  very  name,  the  Cost  Performance  Report  is  a 
performance  report.  Of  course,  the  performance  addressed  in 
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the  CPR  concerns  the  work  related  to  the  contract  itself. 
This  thesis  does  not  suggest  that  providing  feedback  to 
contractors  might  help  the  contract  performance.  However, 
the  utility  to  the  Government  of  the  information  concerning 
contract  performance,  specifically  included  in  the  variance 
analysis  section  of  the  CPR,  could  reasonably  be  expected  to 
improve  by  the  introduction  of  feedback.  Telling  contrac¬ 
tors  how  to  prepare  the  CPR,  via  the  Data  Item  Descriptions 
is  not  enough.  If  Government  personnel  inform  contractors 
what  the  Government  does  with  the  information  once  it  is 
received,  the  utility  of  the  CPR  should  improve. 
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Much  of  the  research  in  the  early  part  of  the  thesis 
centered  on  formulating  the  best  possible  approach  to  answer 
the  investigative  questions  posed  in  Chapter  I: 

1)  What  specific  background  information  do  high-level 
decision-makers  use  in  making  their  decisions? 

2)  How  should  this  information  be  formatted  to  include 
it  as  feedback  to  preparers  of  CPRs. 

3)  Who  should  serve  as  subjects  for  this  experiment? 

4)  How  does  feedback  improve  the  quality  of  perfor¬ 
mance  reports? 

In  this  section  the  author  will  discuss  the  research 
conducted  to  answer  investigative  questions  1,  2,  and  3. 
Investigative  question  4  was  partially  answered  in  Chapter 
II,  but  the  specific  effect  of  the  feedback  used  in  this  ex¬ 
periment  is  addressed  in  Chapter  IV. 

In  addition  to  the  above,  this  chapter  provides  an  in- 
depth  discussion  of  the  instrument  administered  during  the 
experiment  and  discusses  the  two  phases  of  the  experiment: 
the  rater  calibration  phase  and  the  response  evaluation 
phase . 

Selection  of  the  Feedback 

To  answer  investigative  questions  1  and  2,  research  was 
conducted  at  the  Aeronautical  Systems  Division  (ASD)  at 
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Wright-Patterson  AFB  to  determine  what  information  the 
Government  requires  from  contractors  on  major  contracts  and 
in  what  format  this  same  information  is  transmitted  to  upper 
levels  of  the  Government.  ASD  was  chosen  for  this  research 
because  of  its  large  volume  of  contracts  requiring  C/SCSC 
and  its  proximity  and  accessibility  to  the  researcher. 

Information  originating  at  the  contractor’s  facility 
and  reported  to  the  Government  via  the  CPR  is  transcribed 
into  many  different  formats  and  included  in  a  number  of 
reports.  These  reports  are  sent  to  offices  representing 
various  levels  of  the  Government  acquisition  process. 
Appendix  E  is  a  matrix  which  outlines  the  reporting  require¬ 
ments  for  ASD. 

The  Selected  Acquisition  Report  (SAR)  and  Unit  Cost 
Report  (UCR)  are  primarily  used  to  provide  information  to 
Congress  on  the  status  of  funding  requirements  and  overall 
performance  of  major  defense  contracts.  Department  of 
Defense  Directive  7000.3  is  the  guidance  for  preparing  these 
reports. 

The  Defense  Acquisition  Executive  Summary  (DAES)  pro¬ 
vides  similar  contract  information  as  the  two  reports  above, 
but  the  information  is  presented  in  "formats,"  or  sections, 
with  each  format  addressing  a  particular  aspect  of  the 
contract.  The  ultimate  destination  of  the  DAES  is  the 
Office  of  the  Under  Secretary  of  Defense  for  Acquisition. 
Department  of  Defense  Directive  5000.5  provides  guidance  for 
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preparing  the  OAES.  A  scaled-down  version  of  the  DAES  is 
the  Acquisition  Information  Monthly  Report  (AIMR),  which  is 
required  by  policy  letter.  It  requests  only  certain  parts 
of  the  DAES,  but  on  a  monthly  basis  rather  than  quarterly  as 
is  the  case  with  the  DAES. 

The  Acquisition  Program  Baseline  (APB)  and  Program 
Director’s  Acquisition  Report  (PDAR)  provide  much  of  the 
same  information,  but  they  only  go  as  far  as  the  Air  Force 
Systems  Command  level. 

Although  CPR  information  is  an  integral  part  of  each  of 
the  reports  addressed  in  Appendix  E,  in  assessing  the  use¬ 
fulness  of  these  reports  to  this  thesis,  it  was  evident  that 
one  of  the  DAES  formats,  specifically  format  11,  would  serve 
as  the  best  source  of  feedback  to  provide  to  contractors. 

The  information  contained  on  format  11  was  found  in  several 
of  the  reports  and  was  of  major  importance  in  the  PDAR  which 
appeared  to  be  the  most  widely  circulated  report.  This 
format  requires  a  narrative  explaining  any  conditions  which 
are  less  than  satisfactory.  The  narrative  must  address  the 
following:  1)  Problem,  2)  Impact,  3)  Action  Taken,  and  4) 

Get  Well  Date. 

Since  the  Government  is  required  to  submit  the  contract 
information  using  the  DAES  formats,  requiring  the  contrac¬ 
tors  to  submit  the  same  type  of  information  would  make  the 
government  analyst’s  job  much  easier.  This  requirement  does 
not  add  any  extra  work  for  the  contractor  since  the  intent 
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of  variance  analysis  is  to  receive  this  very  same  informa¬ 
tion  (5:18). 

An  example  of  DAES  Format  11  was  included  in  the  feed¬ 
back  version  of  the  experimental  instruments  administered  to 
the  subjects.  The  format  will  be  discussed  in  the  section 
which  explains  the  experimental  instruments. 

Selection  of  the  Subjects 

In  an  informal  poll  of  graduate  students  who  were  con¬ 
ducting  experiments  or  surveys  for  their  theses  efforts  at 
the  Air  Force  Institute  of  Technology  (AFIT),  School  of 
Systems  and  Logistics,  it  was  found  that  a  majority  of  them 
used  graduate  or  Professional  Continuing  Education  (PCE) 
students  as  subjects.  Discussions  with  the  academic  advisor 
for  this  effort,  LTC  Anthony  Presutti ,  confirmed  that  using 
students  as  surrogates  for  the  general  contracting  community 
would  be  val id. 

Permission  was  requested  in  early  March  of  1989,  to  use 
AFIT  students  in  the  PCE  course  SYS  362  (99C)  -  Cost/Sched¬ 
ule  Control  Systems  Criteria  -  as  subjects  for  this  experi¬ 
ment.  Approximately  40-50  students  attend  this  course  which 
meets  four  times  during  the  year.  Generally,  it  is  com¬ 
prised  exclusively  of  Government  personnel.  Occasionally,  a 
few  contractors  attend  the  course. 

Members  of  this  class  are  typically  familiar  with 
variance  analysis,  but  are  not  experts  in  the  field.  The 
experiment  was  designed  to  use  subjects  who  had  a  some 
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familiarity  with  the  subject  area.  It  was  felt  that  schedu¬ 
ling  the  experiment  in  the  latter  part  of  the  class  would 
ensure  that  all  students  became  familiar  with  the  subject 
matter.  Class  89C  met  from  01  May  89  to  19  May  89.  The 
instrument  was  administered  on  17  May  1989  to  a  total  of  40 
PCE  students,  3  of  whom  were  contractor  personnel. 

During  subsequent  discussions  with  people  knowledgeable 
in  this  area,  it  was  determined  that  general i zabi 1 i ty  of  the 
experimental  results  to  the  contracting  community  would  be 
much  easier  to  substantiate  if  contractors  could  be  used  as 
subjects.  After  several  telephone  discussions  with  poten¬ 
tial  sources  for  contractor  involvement  in  this  thesis,  Mr. 
Joe  Houser,  Chairman  of  the  Management  Systems  Subcommittee 
of  the  National  Security  Industrial  Association  (NSIA),  was 
contacted.  He  offered  a  way  to  include  interested  contrac¬ 
tor  personnel  in  this  thesis  by  allowing  the  experiment  to 
be  administered  during  one  of  the  periodic  NSIA  national 
conferences . 

The  original  plan  was  to  try  to  conduct  the  experiment 
using  contractor  personnel  sometime  after  the  scheduled  PCE 
experiment.  However,  Mr.  Houser  presented  a  better  alterna¬ 
tive  by  suggesting  that  the  best  forum  for  administering  the 
experiment  to  contractor  personnel  would  be  the  NSIA  nation¬ 
al  conference  scheduled  for  14-15  March  1989  in  Long  Beach, 
CA.  The  agenda  at  this  conference  was  specifically  designed 
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to  cover  topics  involving  C/SCSC.  The  instrument  was  ad¬ 
ministered  on  15  March  1989  to  55  contractor  personnel. 


Experimental  Design:  The  Instrument 

A  controlled  experiment,  in. which  a  randomly-assigned 
experimental  group  received  a  feedback  version  of  the  in¬ 
strument  and  a  randomly-assigned  control  group  was  given  a 
non-feedback  version  of  the  same  instrument,  was  adminis¬ 
tered  on  two  separate  occasions  to  two  distinct  groups. 

This  segment  of  the  thesis  provides  an  in-depth  discussion 
of  the  instruments  given  at  the  two  locations. 

Additionally,  this  segment  addresses  the  process  used 
in  evaluating  the  responses  received  during  the  two  experi¬ 
ments.  The  rating  process  was  conducted  in  two  phases. 

Phase  one  involved  the  rater  calibration  process.  Phase  two 
dealt  with  the  actual  rating  of  the  responses  received 
during  the  experiments. 

Experiment  at  NSIA  Conference.  On  15  March  1989,  the 
instrument  included  as  Appendix  A  was  administered  to  55 
contractor  personnel  at  the  NSIA  conference  held  in  Long 
Beach,  California  on  14-15  March  1989.  The  experiment  began 
with  a  five  minute  pre-brief  introducing  the  experimenter 
and  the  instrument.  The  two  versions  of  the  instrument  were 
randomly  assigned  to  those  present.  Based  on  conversations 
with  the  thesis  advisor,  20  minutes  was  considered  an  ap¬ 
propriate  amount  of  time  to  complete  the  instrument  and  was 


allotted  to  the  sub.jscts  for  that  purpose.  There  was  a  five 
minute  post-brier  immediately  following  the  experiment. 

Page  1  of  Appendix  A  highlights  the  information  con¬ 
cerning  the  specifics  of  the  experimental  setting.  The 
first  item  of  interest  on  that  page  is  that  only  46  of  the 
55  instruments  were  returned.  There  is  no  explanation 
concerning  the  nine  missing  instruments.  Of  the  46  instru¬ 
ments  returned,  only  42  were  considered  usable.  Of  the 
total  usable  responses,  21  were  responses  on  non-feedback 
versions  and  21  were  responses  on  feedback  versions.  The 
four  unusable  responses  were  either  incoherent  or  simply  did 
not  exist.  Only  one  subject  commented  that  there  was  not 
enough  time  to  complete  the  instrument. 

Pages  2  through  9  of  Appendix  A  are  the  non-feedback 
version  of  the  instrument.  Page  2  introduces  an  imaginary 
contractor  scenario  which  calls  for  the  prompt  action  of  the 
subject.  The  subject  is  provided  some  information  consist¬ 
ing  of  the  company’s  CPR  Format  1  and  internal  variance 
analyses  for  work  breakdown  structure  level  two  items  which 
are  currently  breaching  contract  thresholds.  The  task 
involves  the  writing  of  a  level  one  variance  analysis  on 
Page  7 . 

Page  3  is  the  CPR  Format  1.  Pages  4  through  6  are  the 
internal  variance  analyses  for  the  elements  breaching  the 
thresholds.  Page  7  is  the  blank  page  on  which  the  subjects 
were  to  write  their  level  one  variance  analysis.  Finally, 
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page  8  is  a  demographic  survey  included  to  gain  information 
concerning  the  experience  of  each  subject  with  C/SCSC  and 
with  information  relative  to  the  Program  Director’s  Assess¬ 
ment  Report  (this  issue  will  be  discussed  further  in  the 
next  section. ) 

Pages  10  through  18  of  Appendix  A  are  the  feedback  ver 
Sion  of  the  instrument.  All  of  the  pages  in  this  version 
are  identical  to  the  non-feedback  version  except  for  the 
wording  in  the  introduction  page  of  the  instrument.  Addi¬ 
tionally,  the  feedback  version  includes  an  extra  page  not 
found  in  the  non-feedback  version. 

The  differences  on  the  introduction  page  are  only  to 
discuss  the  feedback  loop  that  has  been  started  and  to 
explain  why  the  extra  page  is  present  in  the  instrument. 

The  extra  page,  found  on  page  7  of  the  feedback  version,  is 
the  feedback  for  this  experiment.  This  page  is  simply  a 
section  of  the  previous  month’s  PDAR  (DAES  Format  11)  out¬ 
lining  the  cost  performance  on  the  imaginary  contract.  As 
in  the  PDAR,  the  variance  is  explained  using  the  categorize 
tion  which  was  explained  previously.  Additionally,  the 
"program  analyst"  has  added  his  own  comments  to  the  report 
to  provide  feedback  to  the  contractor.  The  effect  of  this 
additional  information  was  the  basic  thesis  question.  The 
exact  nature  of  the  rating  process  is  addressed  in  the 
section  on  the  rating  process. 
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Experiment  at  AFIT  C/SCSC  Course.  On  17  May  1989,  the 


instrument  included  as  Appendix  B  was  administered  to  40 
students  (three  of  these  students  were  contractor  personnel) 
during  the  SYS  362  (89C)  class  of  the  Cost/Schedule  Control 
Systems  Criteria  course  held  at  the  Air  Force  Institute  of 
Technology  School  of  Systems  and  Logistics  from  1  May  1989 
to  19  May  1989.  As  in  the  NSIA  experiment,  this  experiment 
began  with  a  five  minute  pre-brief  introducing  the  experi¬ 
menter  and  the  instrument.  The  two  versions  of  the  instru¬ 
ment  were  randomly  assigned  to  those  present.  These  sub¬ 
jects  were  also  given  20  minutes  to  complete  the  instrument 
and  then  post-briefed  for  5  minutes  immediately  following 
the  experiment. 

Page  1  of  Appendix  B  highlights  the  information  con¬ 
cerning  the  specifics  of  the  experimental  setting.  Of  the 
40  instruments  administered,  all  40  were  returned.  There 
were  a  total  of  32  usable  responses.  Of  these,  16  were  from 
the  control  group  and  16  were  from  the  feedback  group. 

There  was  a  larger  proportion  of  unusable  responses  in  this 
group  in  that  8  responses  had  to  be  disregarded. 

The  biggest  problem  was  that  four  subjects  were  unable 
to  put  themselves  in  the  shoes  of  the  contractor  and  instead 
approached  the  variance  analysis  from  the  Government  per¬ 
spective.  There  were  two  incoherent  responses  and  two  sub¬ 
jects  commented  that  there  was  not  enough  time  to  complete 
the  instrument. 
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The  rest  of  the  instrument  is  identical  to  that  given 
at  the  NSIA  conference  except  that  all  the  dates  were  moved 
forward  two  months  to  give  the  instrument  the  appearance  of 
being  current. 

Experimental  Design:  The  Rating  Process 

The  quality  of  the  CPR  was  rated  by  three  experts  in 
the  field  of  C/SCSC.  The  experts  for  this  experiment  were: 
LTC  Tom  Bowman,  ASD/ACCM;  Maj  Ron  Cohen,  SD/ACCI;  and  Mr. 
Charlie  Gardella  (with  assistance  from  Mr.  Art  Marangus), 
ESD/ACCI.  All  three  of  these  individuals  run  the  Perfor¬ 
mance  Measurement  Divisions  at  the  major  procurement  centers 
of  the  Air  Force  Systems  Command. 

Due  to  the  number  of  responses  requiring  a  rating,  it 
was  necessary  to  solicit  the  help  of  more  than  one  rater  in 
order  to  spread  the  responses  into  manageable  amounts.  As 
it  turned  out,  the  spreading  of  the  evaluation  duties  to 
these  three  individuals  resulted  in  the  receipt  of  ratings 
from  a  cross-section  of  the  air  force  product  divisions. 

In  this  section,  the  two  phases  of  the  rating  process 
are  discussed.  The  first  phase  of  the  rating  process  was 
the  rater  calibration  phase  and  the  second  phase  was  the 
response  evaluation  phase. 

The  Rater  Calibration  Phase.  During  the  preliminary 
discussions  concerning  the  method  of  rating  the  responses, 
it  was  determined  that  some  form  of  rater  calibration  must 
occur  to  ensure  that  the  raters  rated  on  an  equal  basis. 
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Appendix  C  is  the  rater  calibration  package  which  was 
sent  to  all  three  expert  raters.  Page  1  of  this  appendix  is 
simply  the  cover  letter  sent  along  with  the  instructions  and 
sample  variances.  Page  2  of  the  appendix  is  the  set  of 
instructions  concerning  the  rater  calibration  phase.  In 
short,  each  rater  was  to  rank  order  the  three  sample  var¬ 
iance  analyses  by  giving  a  rating  of  "1"  to  the  best  and  so 
on . 

All  three  raters  rank-ordered  the  sample  variances  in 
the  same  order.  This  agreement  was  enough  to  conclude  that 
the  three  expert  raters  had  been  calibrated.  PacjC  3  of 
Appendix  C  shows  the  scores  given  by  each  rater  to  the 
sample  variances  analyses. 

The  Response  Evaluation  Phase.  Once  the  three  expert 
raters  had  been  successfully  calibrated  and  all  the  data 
from  the  two  experiments  had  been  tabulated  into  a  format 
amenable  to  rating,  the  next  step  was  to  have  the  experts 
assign  ratings  to  the  responses. 

Each  rater  was  randomly  assigned  one-third  of  the 
usable  results.  Appendix  D  is  the  response  evaluation 
package.  Page  1  of  this  appendix  is  the  cover  letter  sent 
to  the  raters.  Pages  2  and  3  provide  guidance  concerning 
the  experiment  and  the  rating  process.  Beginning  with  page 
4  and  ending  with  page  37,  the  responses  are  listed  under  a 
Likert  scale  heading  and  the  response  ratings  are  included. 
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This  chapter  discusses  the  analysis  of  the  data  ob¬ 
tained  from  the  experiments  conducted  during  this  thesis. 

The  result  of  the  hypothesis  test  used  to  evaluate  the  data 
is  explained  and  all  the  significant  variables  included  in 
the  experiments  are  examined  in  order  to  address  the  vali¬ 
dity  of  the  experimental  instruments  and  response  evaluation 
procedure. 

Hy.pQthe.gjg  TQ$t 

A  hypothesis  test  was  done  to  determine  if  there  was 
any  statistically  significant  difference  in  the  mean  of  the 
feedback  variance  analyses  received  and  the  mean  of  the  non¬ 
feedback  variance  analyses  received.  The  confidence  level 
for  determining  statistical  significance  in  this  thesis  was 
set  at  9056.  There  were  37  samples  in  each  category. 

Appendix  G  provides  the  specific  information  concerning 
the  hypothesis  test  of  the  data  and  gives  the  result  of  the 
test.  As  stated  in  this  appendix,  giving  subjects  feedback, 
via  the  Defense  Acquisition  Executive  Summary  (DAES)  format 
11,  did  not  produce  a  statistically  significant  difference 
in  the  quality  of  the  variance  analyses  as  rated  by  the 
expert  raters. 

As  addressed  in  Chapter  II  of  this  thesis,  the  use  of 
feedback  has  had  a  positive  effect  in  various  settings. 


Providing  feedback  to  contractors  helps  to  improve  the 
communication  between  the  government  and  the  contracting 
community,  thereby  enhancing  the  government  to  contractor 
relationship.  The  lack  of  a  statistical  difference  in  the 
quality  of  the  responses  obtained  during  this  experiment 
does  not  diminish  the  benefits  that  would  be  gained  by 
providing  feedback  to  contractors. 

In  order  to  adequately  analyze  the  hypothesis  test 
result,  it  is  necessary  to  examine  the  significant  variables 
included  in  the  experimental  and  response  evaluation  seg¬ 
ments  of  this  thesis.  The  three  most  significant  variables 
-  the  raters  (and  response  rating  process),  the  subjects, 
and  the  feedback  itself  -  will  be  investigated.  The  purpose 
of  these  investigations  is  to  determine  if  the  variables 
were  handled  appropriately  during  the  experimental  and 
response  evaluation  segments  of  the  thesis  and  to  offer  some 
possible  explanations  for  the  lack  of  a  statistically  signi¬ 
ficant  difference  in  the  quality  of  the  subject  variance 
analyses  received. 

Raters  and  Rating  Process 

For  the  purpose  of  this  thesis,  the  expert  raters  had 
complete  anonymity  concerning  the  ratings  they  provided  for 
the  responses.  Appendix  F  tabulates  the  scores  for  each  re¬ 
sponse  and  categorizes  the  raters  only  as  rater  A,  rater  B, 
and  rater  C. 
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In  order  to  diminish  rater  error,  the  raters  were 
calibrated  during  the  first  part  of  the  response  evaluation 
phase.  As  addressed  in  Chapter  III,  this  process  was  done 
to  demonstrate  that  the  raters  were  consistent  in  their 
rating  of  variance  analyses  and  to  allow  the  author  to 
divide  the  responses  into  manageable  portions. 

Additionally,  as  tabulated  in  Appendix  F,  the  raters 
were  randomly  assigned  both  feedback  and  non-feedback  re¬ 
sponses  from  all  the  responses  obtained.  This  randomization 
helped  to  further  suppresses  rater  error. 

A  series  of  t- tests  were  conducted  to  address  the  issue 
of  statistically  significant  differences  between  the  raters. 
In  addition,  t- tests  were  conducted  to  investigate  whether 
there  was  a  statistically  significant  difference  in  the 
feedback  to  non-feedback  scores  within  each  rater. 

Differences  Between  Raters.  Six  separate  t-tests 
addressing  the  differences  between  raters  were  conducted. 
Feedback  ratings  for  each  rater  were  individually  compared 
to  the  feedback  ratings  of  the  other  raters.  Similarly, 
non-feedback  ratings  for  each  rater  were  individually  com¬ 
pared  to  the  non-feedback  ratings  of  the  other  raters.  In 
all  six  cases,  no  statistically  significant  difference  was 
found  between  raters. 

The  lack  of  a  statistically  significant  difference 
between  raters  supports  the  notion  that  the  raters,  as  a 
whole,  rated  the  feedback  responses  equally  as  compared  to 
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the  feedback  ratings  of  the  other  raters.  The  same  logic 
holds  true  for  the  rating  of  the  non-feedback  responses 
between  raters.  Until  one  compares  the  feedback  to  non¬ 
feedback  responses,  the  only  assertion  that  can  be  made  thus 
far  is  that  the  raters  are  consistent  in  their  rating  of 
feedback  and  non-feedback  responses. 

Differences  Within  Raters.  Three  t-tests  were  con¬ 
ducted  to  investigate  the  individual  differences  in  the 
feedback  to  non-feedback  ratings  within  each  rater.  In  all 
three  cases,  no  statistically  significant  differences  were 
found. 

Combining  the  lack  of  statistically  significant  dif¬ 
ferences  within  each  rater  with  the  lack  of  a  statistically 
significant  difference  overall,  results  in  one  of  two  con¬ 
clusions.  The  first  is  that  there  is  no  difference  in  the 
quality  of  variance  analysis  when  one  provides  feedback,  as 
given  in  these  experiments,  to  individuals  who  prepare  the 
CPR.  (This  conclusion  will  be  addressed  in  the  next  chap¬ 
ter.)  The  other  conclusion  is  that  some  confounding  vari¬ 
able  exists  which  may  be  affecting  the  results. 

Possible  Confounding  Variables  in  the  Rating  Process. 
Durso  and  Mellgren  discuss  a  number  of  confounding  variables 
which  can  affect  experimental  results  (8:84-86).  Two  of 
these  variables,  "history"  and  "instrumentation,"  are  rele¬ 
vant  in  analyzing  the  rating  process  of  this  thesis,  speci¬ 
fically  as  it  pertains  to  the  raters  themselves. 
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As  noted  in  Appendix  C,  the  calibration  packages  were 
sent  to  the  three  expert  raters  on  12  May  89.  Raters  A  and 
C  returned  the  package  within  a  week,  while  rater  B  took 
about  three  weeks  to  return  the  package.  A  similar  time  lag 
was  experienced  by  rater  B  in  returning  the  response  evalua¬ 
tion  package  (Appendix  0)  mailed  on  12  June  89.  While 
raters  A  and  C  took  less  than  two  weeks  to  return  the  re¬ 
sponses,  rater  B  needed  over  two  months  to  complete  the 
evaluations. 

Durso  and  Mellgram  define  “history"  as,  "specific 
events  external  to  the  subject  that  distinguish  one  condi¬ 
tion  from  another  besides  the  independent  variable"  (8:85), 
In  the  rating  process,  the  "subject"  mentioned  in  the  def¬ 
inition  would  be  each  rater  himself.  It  is  possible  that  the 
rater’s  evaluations  may  have  been  affected  differently  by 
their  surroundings. 

The  confounding  variable,  "instrumentation,"  is  defined 
as  "changes  in  the  instrument  used  to  measure  the  dependent 
variable"  (8:85).  It  is  possible  that  this  confounding 
variable  may  also  have  affected  the  response  evaluation 
since  different  raters  rated  the  responses.  Randomization 
and  rater  calibration  were  specifically  intended  to  control 
for  this  confounding  variable,  but  the  possibility  exists 
that  instrumentation  still  affected  the  response  evaluation 
phase  of  this  thesis. 
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Experimental  Sub.iects 

The  validity  of  using  the  Professional  Continuing 
Education  (PCE)  students  as  subjects  for  the  experiments  was 
discussed  in  Chapter  III.  The  general izabi 1 ity  of  the 
experimental  results  was  enhanced  by  also  using  contractor 
personnel  as  subjects.  For  the  purpose  of  this  thesis,  the 
two  groups  were  considered  together  and  no  comparison  was 
made  between  them. 

Although  a  few  of  the  responses  received  from  both 
groups  were  unusable,  their  existence  did  not  affect  the 
experiment.  A  satisfactory  amount  of  responses  were  ob¬ 
tained  to  conduct  the  "large  sample"  hypothesis  test  out¬ 
lined  in  Appendix  G. 

The  type  of  instrument  (feedback  or  non-feedback)  was 
randomly  assigned  to  each  subject.  The  only  differences  in 
the  two  instruments  were  outline  in  Chapter  III  and  dealt 
with  changing  the  scenario  dates  to  make  the  instruments 
appear  up-to-date. 

The  same  experimental  procedure  was  used  in  each  ex¬ 
periment.  Subjects  were  briefed  by  the  experimenter  using 
the  same  briefing  in  both  experiments.  The  only  difference 
between  the  two  cases  was  the  title  page  which  noted  the 
location  of  the  experiment 

In  the  absence  of  any  other  unknown  variables,  no  con¬ 
founding  errors  concerning  the  subjects  were  noted  during 
the  conduct  of  either  experiment. 
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Experimental  Feedback 


As  noted  in  Appendix  G  and  previously  mentioned  in  the 
section  on  the  rating  process,  there  was  no  statistically 
significant  difference  in  the  overall  feedback  to  non-feed¬ 
back  ratings,  nor  was  there  a  statistically  significant 
difference  in  the  individual  rater  feedback  to  non-feedback 
ratings.  This  section  analyzes  the  feedback  as  given  in  the 
experiments  to  determine  if  any  confounding  variables  in 
this  area  may  have  affected  the  response  evaluations. 

Content  of  Feedback.  Chapter  III  addresses  the  pre¬ 
liminary  research  conducted  in  order  to  assess  the  feedback 
potential  of  various  types  of  government  reports.  The 
rationale  given  for  selecting  Format  11  of  the  Defense 
Acquisition  Executive  Summary  (DAES)  as  the  feedback  to  use 
in  the  experiments  was  that  this  format  seemed  to  be  the 
most  widely  distributed  of  the  reports  examined. 

The  sub-categories  included  in  this  format  outline  the 
type  of  information  which  the  government  requires  in  order 
to  report  to  higher  levels  of  the  acquisition  community. 

The  assumption  that  the  most  widely  distributed  report 
format  would  provide  the  best  source  of  feedback  to  subjects 
may  not  have  been  totally  correct.  Many  other  reports 
provide  the  same  type  of  information  to  different  levels  of 
the  government  acquisition  environment.  Perhaps  another 
report  would  provide  a  better  source  of  feedback.  Addi¬ 
tional  research  in  this  area  is  appropriate. 
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Feedback  and  Time  Allotted.  It  is  also  possible  that  a 


more  simple  source  of  feedback  would  work  better  in  the 
experimental  setting.  Although  subjects,  with  very  few 
exceptions,  had  enough  time  to  complete  the  experiments,  the 
feedback  group  was  given  the  same  amount  of  time  to  complete 
the  experiment  as  was  given  to  the  non-feedback  group. 

It  is  possible  that  including  an  entire  page  of  infor¬ 
mation  affected  the  quality  of  the  responses  given  by  the 
feedback  groups.  Since  the  feedback  groups  had  to  assimi¬ 
late  the  feedback  information,  they  had  less  time  to  provide 
a  response.  Perhaps  providing  a  form  of  feedback  which 
requires  less  time  to  digest  might  help  to  reduce  this 
possible  confounding  variable. 

Presentation  of  Feedback.  In  order  to  maintain  the 
integrity  of  the  experimental  setting,  the  feedback  subjects 
were  not  given  any  additional  information  concerning  the 
feedback  other  than  what  was  given  in  the  instruments.  It 
is  possible  that  simply  placing  the  feedback  in  the  instru¬ 
ments  was  not  enough.  Had  the  subjects  received  a  briefing 
on  the  feedback  itself,  or  had  they  been  allowed  to  discuss 
the  feedback,  the  results  might  have  been  different. 

In  an  actual  government/contractor  setting,  the  com¬ 
munication  and  feedback  loop  is  more  highly  developed. 
Government  analysts  and  contractor  personnel  have  frequent 
discussions  concerning  the  quality  of  the  data  included  in 
the  CPRs.  Additionally,  contractors  have  ample  time  to 
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consider  the  feedback  provided  by  government  analysts  and 
determine  how  to  incorporate  this  information  into  the 
variance  analyses  they  include  in  their  CPR  submissions. 

The  additional  considerations  noted  above  may  not  have 
been  adequately  addressed  by  simply  including  a  page  of 
comments  as  feedback  within  an  experimental  instrument  and 
giving  subjects  20  minutes  to  incorporate  this  information 
into  their  responses. 
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This  thesis  is  the  first  attempt  at  improving  the 
utility  of  the  Cost  Performance  Report  by  addressing  issues 
at  the  "grass  roots"  level.  Given  the  discussion  of  pos¬ 
sible  confounding  variables  in  the  previous  chapter,  it  is 
not  possible  to  reach  a  definite  conclusion  concerning  the 
effect  of  feedback  on  Cost  Performance  Report  (CPR)  utility 
as  studied  in  this  thesis. 

This  chapter  discusses  the  results  of  the  analysis 
included  in  the  previous  chapter.  Depending  on  the  reader’s 
interpretation  of  the  possible  confounding  variables  dis¬ 
cussed  in  Chapter  IV,  two  conclusions  are  possible.  These 
conclusions  are  addressed  in  this  chapter.  Recommendations 
concerning  follow-up  research  in  this  area  are  given  for 
each  of  the  conclusions. 

Conclusion  A: _ Experiment  Inconclusive 

Chapter  IV  addressed  the  areas  in  which  possible  con¬ 
founding  variables  may  exist.  The  three  main  areas  of 
discussion  are:  the  rating  process;  the  subjects;  and  the 
feedback.  If  the  reader  believes  in  the  existence  of  one  of 
the  confounding  variables  mentioned,  or  perhaps  in  one  not 
mentioned,  then  the  conclusion  of  this  thesis  is  that  the 
experiment  is  inconclusive. 
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The  author’s  recommendation  is  to  run  the  experiment 
again  and  control  for  the  confounding  variables  which  are 
believed  to  exist.  Suggested  further  control  depends  on  the 
area  of  the  experiment  where  the  confounding  variables  are 
believed  to  exist. 

Rating  Process  Control.  The  raters  in  this  thesis  were 
not  given  a  definite  time  in  which  to  complete  either  of  the 
two  phases  of  response  evaluation.  The  expert  raters  took 
as  much  time  as  they  felt  necessary  to  complete  and  return 
both  the  calibration  and  response  evaluation  packages. 

To  control  for  potential  errors  in  “history"  as  defined 
in  the  previous  chapter,  the  author  recommends  that  raters 
be  placed  under  the  same  type  of  controlled  setting  as  the 
subjects  were.  Perhaps  it  would  be  possible  to  gather  the 
expert  raters  in  one  room  and  allow  them  only  a  certain 
amount  of  time  to  complete  the  calibration  and  response 
evaluation  packages. 

To  completely  control  for  potential  errors  in  "instru¬ 
mentation"  as  defined  in  the  previous  chapter,  the  author 
recommends  that  one  expert  rater  be  given  the  task  of  rating 
all  of  the  evaluations.  Although  randomization  and  calibra¬ 
tion  were  intended  to  control  for  this  confounding  variable, 
having  one  expert  rater  rate  the  responses  eliminates  any 
possibility  that  this  confounding  variable  will  affect  the 
results. 
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necessary . 

Possible  changes  to  the  feedback  given  during  this 
thesis  include,  but  are  not  limited  to:  1)  Changing  the 
form  in  which  the  feedback  is  given;  2)  Increasing  the 
amount  of  time  allotted  to  feedback  subjects  for  completion 
of  the  instrument;  and  3)  Providing  a  feedback  briefing  in 
conjunction  with  the  feedback  given  in  the  instruments. 

Enhancing  the  prominence  of  the  feedback  by  using 
more  direct  vocabulary  might  result  in  a  statistically 
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significant  difference  in  the  quality  of  the  responses.  For 
example,  the  wording  in  the  instrument  might  read,  "this  in¬ 
formation  is  of  vital  importance,"  instead  of,  "this  infor¬ 
mation  is  recommended." 

Conclusion  B:  Feedback  Has  No  Effect  on  CPR  Utility 

Until  now,  all  discussions  in  the  two  last  chapters 
have  centered  on  possible  confounding  variables  in  the 
experiments  which  may  have  affected  the  hypothesis  test  and 
led  to  an  incorrect  conclusion.  This  approach  is  partially 
based  on  a  methodical  analysis  of  the  assumptions  used  in 
this  thesis.  Admittedly,  the  intuitive  belief  that  feedback 
should  have  a  positive  effect  on  CPR  utility  also  fueled  the 
author’s  critical  approach  towards  the  experimental  proce¬ 
dure  used  in  this  thesis. 

However,  if  one  accepts  the  assumptions  of  this  thesis, 
the  conclusion  that  feedback  (as  given  in  this  experiment) 
has  no  effect  on  CPR  utility  is  completely  logical.  Perhaps 
the  additional  information  provided  by  the  feedback  is  not 
necessary.  It  may  be  that  individuals  who  prepare  the  CPR 
are  familiar  with  the  needs  of  the  Government. 

Government  guidance  documents  are  very  specific  in 
outlining  how  contractors  should  prepare  government  reports. 
Although  the  literature  suggests  that  contractors  could 
improve  the  quality  of  the  CPR  variance  analyses,  perhaps 
the  way  to  do  so  does  not  involve  the  use  of  feedback. 
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Appendix  A:  NSIA  Survey  (Page  1  of  18) 


National  Security  Industrial  Association  Survey 
Long  Beach,  California 
March  15,  1989 


Subjects :  Contractors  at  all  levels  of  the  organization 
with  very  few  exceptions,  all  of  the  subjects  had  an 
extensive  knowledge  of  C/SCSC.  Most  had  at  least  a  moderate 
level  of  working  with  the  Cost  Performance  Report  and  a 
moderate  level  of  writing  variance  analysis  reports. 


Statistics:  55  Administered 

46  Returned 
42  Usable 
21  Control  Group 
21  Experimental  Group 


Errors:  I  became  aware  of  two  errors  in  the  instrument. 

One  involved  the  inadvertent  switching  of  two  variance 
explanations.  The  group  was  briefed  on  this  error. 
Additionally,  one  of  the  subjects  came  to  me  after  the 
experiment  noting  a  math  error  in  one  of  the  columns.  This 
column  was  not  a  key  area  of  the  experiment.  (Though 
neither  error  was  consequential,  this  paragraph  was  added 
to  note  that  the  errors  were  noticed. ) 
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Appendix  A:  NSIA  Survey  (Page  3  of  18) 


Page  2 


INTRODUCTION 


Scenario:  Ms.  Lee,  the  JHA  SYSTEMS  Program  Control  Manager 
on  the  DAGGER  F-26  program,  is  away  on  an  emergency. 

Although  you  are  very  busy  as  a  Program  Control  Manager  on 
another  program,  the  F-26  Program  Manager,  Mr.  Jones,  has 
requested  your  help  since  it  appears  Ms.  Lee  will  be 
unavailable  for  a  few  weeks.  Today  is  the  thirteenth 
working  day  after  the  February  accounting  close.  The  CPR  is 
due  to  the  customer  on  the  17th  working  day  following  the 
accounting  close.  In  order  to  meet  your  deadline,  you  must 
finish  the  CPR  today.  The  data  is  in,  but  the  variance 
analysis  is  yet  to  be  completed. 

Information  Available:  Mr.  Phillips,  the  F-26  Assistant 
Program  Control  Manager,  has  brought  you  the  JHA  Systems 
Management  Reports  which  help  explain  the  cost  situation. 
These  reports  are  required  by  Mr.  Jones  for  all  work 
breakdown  structure  elements  which  have  a  cost  variance  in 
excess  of  105»J  for  current  month  or  in  excess  of  555  and 
$250,000  at  cumulative. 

Task:  Your  immediate  task  is  to  use  the  information  in  the 
CPR  and  JHA  Management  Reports  to  write  up  a  variance 
analysis  report  (included  as  page  7  of  this  package)  for  the 
total  contract  cost  variance  only.  Mr.  Phillips  is  handling 
all  the  schedule  variance  analyses  as  well  as  the  level  2 
and  lower  (if  required)  cost  variance  analyses.  The  vari¬ 
ance  analysis  report  you  write  should  meet  with  approval 
from  Mr.  Jones  and  the  government.  You  have  a  meeting  with 
Mr.  Jones  in  15  minutes  to  discuss  the  overall  status  of  the 
F-26  contract  cost  variance.  Good  luck! 


Note:  Format  5  variance  analysis  guidance  states  the 

following  for  total  contract  reporting: 

"Provide  a  summary  analysis,  identifying  significant 
problems  affecting  performance.  Indicate  corrective  actions 
required,  including  Government  action  where  applicable." 


49 


I 


-  V<0(lK  OnCAKOOWN  STHUCTUnE 


5 


Appendix  A:  NSIA  Survey  (Page  5  of  18) 


JHA  SYSTEMS 

i\L4NAGEMENT  REPORT 

i  COST 

□  CURRENT  PERIOD  >+10% 

□  SCHEDULE 

M  CURRENT  PERIOD  >-10% 

□  TECHNICAL 

□  CUMULATIVE  >+5%  k  |250K 

□  CONTRACTS 

S  CUMULATIVE  >- 

5%  k  I250K 

ITBS  ELEMENT:  AIRFRAME 

MANAGER;  R. 

MARSHALL 

CURRENT/(%) 

cum/(%) 

BAC;  I139469K 

COST  VAR  (.|59flK)/(l5,9%) 

(|8416K)/(9.0%) 

EAC:  I153093K 

SCHED  VAR  ( $403 K)/(  10.7%) 

($4223K)/(4.0%) 

NARRATIVE; 


We  are  still  having  difficulty  with  the 
chemical  structure  of  the  composite  material.  The 
problem  surfaces  are  the  vertical  stabilizer  and 
the  trailing  portions  of  the  wings.  The  problem 
is  that  the  exhaust  from  the  engines  is  reacting 
with  the  chemical  composition  of  the  lifting 
surfaces,  forming  a  thin  coating  on  the  surfaces. 
This  residue  is  affecting  the  flight  character¬ 
istics  of  the  lifting  surfaces.  I  have  discussed 
the  problem  with  Mr.  Andersen  and  he  is  looking 
into  possibly  altering  the  exhaust  heat.  We  have 
been  analyzing  the  residue  to  see  how  we  can  alter 
the  chemical  structure  of  the  composite  material 
and  are  close  to  uncovering  the  problem.  I  expect 
resolution  of  this  problem  by  June  with  minimal 
impact  to  the  program. 
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JHA  SYSTEMS  “34/89 

MNAGEMENT  REPORT 

[1]  COST 

□  CURRENT  PERIOD  >+lQ% 

□  SCHEDULE 

E  CURRENT  PERIOD  >-10% 

□  TECHNICAL 

□  CUMULATI\T:  >+5%  k  $250K 

n  CONTRACTS 

El  CUMULATIVE  >-5%  k  5250K 

WBS  ELEMENT;  PROPULSION 

!  MANAGER;  C,  ANDERSEN 

CURRENT/(%)  CUM/(%)  RAC;  I49233K 


COST  VAR  {$11QK)/(10.5%)  (|ia02K)/(5.7%)  EAC:  f5221QK 


SCHED  VAR  (|106K)/(15.1%)  ($2061K)/(5.5%) 


NARRATIV^E: 

We  continue  to  have  a  problem  with  the 
chemical  reaction  between  the  exhaust  and  the 
composite  material.  I  have  tried  altering  the 
exhaust  gas  temperature  (EGT),  but  it  has  not 
affected  the  accumulation  of  sediment  on  the 
vertical  stabilizer  and  trailing  portions  of  the 
wings.  The  changes  in  the  EGT  have  caused  my  WBS 
element  to  incur  the  variances  noted  above.  I 
will  continue  with  my  plans  and  remain  optimistic 
that  the  designed  EGT  is  appropriate.  However,  I 
may  have  to  modify  other  areas  of  the  propulsion, 
hence  my  revised  EAC  is  as  previously  submitted  to 
you  and  noted  above.  The  problem  will  be  resolved 
when  the  AIRFRAME  WBS  Manager  solves  the  chemical 
structure  problem  of  the  composite  material.  My 
discussion  with  Mr,  Marshall  signifies  that  the 
problem  should  be  resolved  by  June. 
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Appendix  A:  NSIA  Survey  (Page  7  of  18) 


JHA  SYSTEMS 

R4NAGEMENT  REPORT 

!S  COST 

□  CURRENT  PERIOD  >+10% 

□  SCHEDULE 

S  CURRENT  PERIOD  >-lQ% 

□  TECHNICAL 

□  CUMirUTIVE  >+5%  &  I250K 

□  CONTRACTS 

M  CUMUUTrVE  >-5%  k  $250K 

ITBS  ELEMENT;  SYS  TEST  & 

E\'Al|  MANAGER;  D.  KTOTO 

CURRENT/(%) 

CUM/(%)  BAC;  127223K 

COST  VAR  (il54K)/(24.5%) 

($1811K)/(9.7%)  EAC;  $29893K 

SCHED  VAR  (|145K)/(23.1%) 

(|202OK)/(14.1%) 

NARIUTi;^: 


The  most  recent  test  suggests  that 
either  the  chemical  composition  of  the  lifting 
surfaces  needs  to  change  or  the  propulsion  exhaust 
gas  temperature  (EGT)  must  change.  Both  WBS 
managers  are  aware  of  the  recurring  problem.  It 
has  set  me  back  as  noted  above.  The  next  systems 
test  is  In  ^uly.  Both  managers  have  assured  me 
that  the  problem  will  be  fixed  by  then  and  that 
the  system  will  pass  the  test.  No  further  impacts 
are  expected,  except  for  the  unfavorable  cost 
variance  at  completion  as  already  recorded  by  the 
Cost  Accounting  System. 
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NSIA  Survey  (Page  9  of  18) 


Page  8 


SURVEY  QUESTIONNAIRE 

Please 

circle  the  most 

appropriate  responses. 

1.  What  is  your  level 

of  education? 

A. 

High  School  Graduate. 

B. 

Some  College. 

C. 

4  Year  College 

Graduate. 

D. 

Masters . 

E. 

Post-Masters . 

2.  What  is  your  level  of  familiarity  with  Department  of 
Defense  Contracts? 

A.  Never  worked  on  one. 

B.  Some  experience  working  with  DoD. 

C.  Moderate  knowledge  of  DoD  practices. 

D.  Extensive  familiarity  with  DoD  practices. 

3.  What  is  your  level  of  experience  with  Cost/Schedule 
Control  Systems  Criteria? 

A.  None. 

B.  Some  knowledge  of  C/SCSC. 

C.  Moderate  knowledge. 

D.  Extensive  familiarity  with  C/SCSC. 

4.  What  is  your  level  of  experience  with  the  Cost 
Performance  Report? 

A.  None. 

B.  Some  knowledge  of  the  CPR. 

C.  Moderate  knowledge  of  the  CPR. 

D.  Extensive  knowledge  of  the  CPR. 

5.  What  experience  do  you  have  writing  Variance  Analyses? 

A.  None. 

B.  Some  experience. 

C.  Moderate  experience. 

D.  Extensive  experience. 

6.  What  is  your  level  of  familiarity  with  the  Program 
Director’s  Assessment  Report? 

A.  I  don’t  know  what  it  is. 

B.  Some  familiarity  with  the  PDAR. 

C.  I  have  moderate  familiarity  with  the  PDAR. 

D.  I  have  extensive  familiarity  with  the  PDAR. 
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Appendix  A:  NSIA  Survey  (Page  11  of  18) 
INTRODUCTION 


Page  2 


Scenario:  Ms.  Lee,  the  JHA  SYSTEMS  Program  Control  Manager 
on  the  DAGGER  F~26  program,  is  away  on  an  emergency.  Al¬ 
though  you  are  very  busy  as  a  Program  Control  Manager  on 
another  program,  the  F-26  Program  Manager,  Mr.  Jones,  has 
requested  your  help  since  it  appears  Ms.  Lee  will  be  un¬ 
available  for  a  few  weeks.  Today  is  the  thirteenth  working 
day  after  the  February  accounting  close.  The  CPR  is  due  to 
the  customer  on  the  17th  working  day  following  the  account¬ 
ing  close.  In  order  to  meet  your  deadline,  you  must  finish 
the  CPR  today.  The  data  is  in,  but  the  variance  analysis  is 
yet  to  be  completed. 

Information  Available:  Mr.  Phillips,  the  F-26  Assistant 
Program  Control  Manager,  has  brought  you  the  JHA  Systems 
Management  Reports  which  help  explain  the  cost  situation. 
These  reports  are  required  by  Mr.  Jones  for  all  work  break¬ 
down  structure  elements  which  have  a  cost  variance  in  excess 
of  10%  for  current  month  or  in  excess  of  5%  and  $250,000  at 
cumulative.  Additionally,  Mr.  Phillips  has  brought  you  a 
copy  of  the  customer’s  report,  based  on  last  month’s  data, 
which  the  customer  sends  to  its  headquarters.  Ms.  Lee 
requested  a  copy  of  the  customer  report  to  understand  what 
the  government  does  with  the  data  it  receives  in  the  CPR. 

The  feedback  Ms.  Lee  has  received  from  the  government 
centers  on  the  lack  of  a  complete  explanation  of  the  problem 
areas.  Capt  Hoyt,  the  customer  F-26  program  control 
analyst,  thought  that  sending  Ms.  Lee  a  copy  of  the  previous 
Program  Director’s  Comments  would  show  her  just  how  much 
detail  was  actually  missing  in  the  variance  analysis.  He 
included  handwritten  comments  on  the  copy  he  sent  Ms.  Lee  to 
point  out  that  the  information  the  government  requests  "up 
the  chain"  was  not  making  its  way  into  the  variance  analysis 
reports,  especially  at  total  contract  level. 

Task:  Your  immediate  task  is  to  use  the  information  in  the 

CPR  and  JHA  Management  Reports  to  write  up  a  variance 
analysis  report  (included  as  page  7  of  this  package)  for  the 
total  contract  cost  variance  only.  Mr.  Phillips  is  handling 
all  the  schedule  variance  analyses  as  well  as  the  level  2 
and  lower  (if  required)  cost  variance  analyses.  The  vari¬ 
ance  analysis  report  you  write  should  meet  with  approval 
from  Mr,  Jones  and  the  government.  You  have  a  meeting  with 
Mr.  Jones  in  15  minutes  to  discuss  the  overall  status  of  the 
F-26  contract  cost  variance.  Good  luck! 

Note:  Format  5  variance  analysis  guidance  states  the 
following  for  total  contract  reporting:  "Provide  a  summary 
analysis,  identifying  significant  problems  affecting 
performance.  Indicate  corrective  actions  required, 
including  Government  action  where  applicable." 
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Appendix  A;  'NSIA  Survey  (Page  13  of  18) 


JHA  SYSTEMS 

.\L4NAGEMENT  REPORT 

g]  COST 

U  CURRENT  PERIOD  >+10% 

□  SCHEDULE 

M  CURRENT  PERIOD  >-10% 

□  TECHNICAL 

□  CUMULATIVE  >+5%  k  i350K 

□  CONTRACTS 

IS  CUMULATIVE  >-5%  6;  S250K 

WBS  ELEMENT:  AIRFRAME 

1  MANAGER;  R.  MARSHALL 

r'lIRRPlSTT  /l%\ 

*«*  A  fA  f  AiAA  1  A  ,/ 

ruM/r^i  Pin- 

COST  VAR  (f599K)/(10.9%) 

($a416K)/(9.e%)  EAC;  I153093K 

SCHED  VAR  ($403K)/(10.7%) 

.($4223K)/(4,8%) 

KARRAmE; 

We  are  still  having  difficulty  with  the 
chemical  structure  of  the  composite  material.  The 
problem  surfaces  are  the  vertical  stabilizer  and 
the  trailing  portions  of  the  wings.  The  problem 
is  that  the  exhaust  from  the  engines  is  reacting 
with  the  chemical  composition  of  the  lifting 
surfaces,  forming  a  thin  coating  on  the  surfaces. 
This  residue  is  affecting  the  flight  character¬ 
istics  of  the  lifting  surfaces.  I  have  discussed 
the  problem  with  Mr.  Andersen  and  he  is  looking 
into  possibly  altering  the  exhaust  heat.  We  have 
been  analyzing  the  residue  to  see  how  we  can  alter 
the  chemical  structure  of  the  composite  material 
and  are  close  to  uncovering  the  problem.  I  expect 
resolution  of  this  problem  by  June  with  minimal 
impact  to  the  program. 
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JHA  SYSTEMS  w 

M4NAGEMENT  REPORT 

12  COST 

□  CURRENT  PERIOD  >+10% 

U  SCHEDULE 

E  CURRENT  PERIOD  >-10% 

□  TECHNICAL 

□  CUMULATIVE  >+5%  «  ?250K 

□  CONTRACTS 

S  CUMUUTIVE  >-5%  k  $250K 

CURRENT/(%)  CUM/(%) 
COST  VAR  ($116K)/(10.5%)  (|l802K)/(5,7%) 
SCHED  VAR  (|igSK)/(l5.1%)  (|2O61K)/(0.5?i) 


NARRATI\’E; 

We  continue  to  have  a  problem  with  the 
chemical  reaction  between  the  exhaust  and  the 
composite  material.  I  have  tried  altering  the 
exhaust  gas  temperature  (EGT),  but  it  has  not 
affected  the  accumulation  of  sediment  on  the 
vertical  stabilizer  and  trailing  portions  of  the 
wings.  The  changes  in  the  EGT  have  caused  my  WBS 
element  to  incur  the  variances  noted  above.  I 
will  continue  with  my  plans  and  remain  optimistic 
that  the  designed  EGT  is  appropriate.  However,  I 
may  have  to  modify  other  areas  of  the  propulsion, 
hence  my  revised  EAC  is  as  previously  submitted  to 
you  and  noted  above.  The  problem  will  be  resolved 
when  the  AIRFRAME  WBS  Manager  solves  the  chemical 
structure  problem  of  the  composite  material.  My 
discussion  with  Mr.  Marshall  signifies  that  the 
problem  should  be  resolved  by  June. 


PAP. 

ijA  r  V  V  Ik 

EAC:  I52213K 


MANAGER:  C.  ANDERSEN  . 


ms  ELEMENT:  PROPULSION 
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Appendix  A: 


Survey  (Page  15  of  18) 


JHA  SYSTEMS 

mAGEMENT  REPOET 


U 

2/24/09 


□  SCHEDULE 
n  TECHNICAL 
n  CONTRACTS 


□  CURRENT  PERIOD  >+10% 

'1  CURRENT  PERIOD  >-10% 

□  CUMUUTIVE  >+5%  k  |250K 
!l  CUMUlATrVT  >-D%  k  i250K 


WBS  ELEMENT:  SYS  TEST  k  E’aii  iL4NACTER:  D.  KTOTO 


V  2  VI V  JJ  A 1  j.  /  \  .*•  ; 


CUM/(%) 


RAr. 


COST  VAR  ^|154K)/(24.5%)  ($1011K)/(9.7%)  EAC:  $29093K 

SCHED  VAR  ($145K)/(23,1%)  ($202OK)/(14.1%) 


NARRATIVE; 

The  most  recent  test  suggests  that 
either  the  chemical  composition  of  the  lifting 
surfaces  needs  to  change  or  the  propulsion  exhaust 
gas  temperature  (EGT)  must  change.  Both  WBS 
managers  are  aware  of  the  recurring  problem.  It 
has  set  me  back  as  noted  above.  The  next  systems 
test  is  in  July.  Both  managers  have  assured  me 
that  the  problem  will  be  fixed  by  then  and  that 
the  system  will  pass  the  test.  No  further  impacts 
are  expected,  except  for  the  unfavorable  cost 
variance  at  completion  as  already  recorded  by  the 
Cost  Accounting  System. 
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Appendix  A: 


(Page  16  of  18) 


♦♦UNCLASSIFIED** 


Page  7 


DAGGER  F-26  March,  1989 

Format  11  -  Program  Director’s  Comments  PREPARED:  02/27/89 


Explanation  of  Y^in  Cost  Performance 


-PROBLEM: 

JHA  Systems  is  experiencing  problems  in  the  airframe.  The 
specific  problem  is  unknown,  but  it  seems  to  be  centered  on 
the  shaping  of  the  composite  materials.  JHA  also  has 
problems  with  the  Propulsion  WBS  element.  The  specific 
problem  is  unknown.  Finally,  JHA  is  falling  behind  in 
System  Test  and  Evaluation. 

-IMPACT: 

There  is  no  indication  that  JHA  will  be  able  to  recover  the 
cost  variance  satisfactorily.  JHA  has  not  specifically 
identified  the  problem  and  does  not  really  know  the  impact 
of  the  problem.  JHA  estimates  a  cost  overrun  of  $7.3 
million,  but  our  estimate  is  a  $15.6  million  overrun. 

-ACTION  TAKEN: 

The  AFPRO  and  DCAS  personnel  have  been  asked  to  look  into 
the  airframe  problem  to  help  determine  the  extent  of  the 
design  shortcomings.  The  program  director  will  meet  the  JHA 
program  manager  to  discuss  the  cost  situation. 

At  -GET  WELL: 

With  AFPRO,  DCAS,  and  JHA  management  attention,  trend 
reversal  expected  Apr  89. 


^  Lee  •.  'i  fcClyj  <1.  "'{gLLOVi"  OSV 

<X.  P»t<vACa;(  . 

V^Ov<0€r  ,  iV  IZ  (k.Vo 

j„v.  Pofti%\A'T  S' 

?Vftiie  W.W-V  retell  V 

IV tK\>  \V«\pf  * 
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JHA  SYSTEMS 

VAEIANCE  ANALYSIS  REPORT 


AS  OF: 
2/24/99 


[Xj  COST 

□  SCHEDULE 

□  TECHNICAL 

□  CONTRACTS 

□  CURRENT  PERIOD  >+10% 

□  CURRENT  PERIOD  >-10% 

□  CUMITATI^T  >+-5%  k  I2-50K 

M  CUMUUTI\T:  >-5%  &  $250K 

WBS  ELEMENT;  TOTAL  CONTRACT 

MANAGER:  R.  JONES 

CURRENT/(%) 

V  V  i*l  /  r  /  -f  !  1.*  A  V  r  V  V  i  V  P  M  AW 

COST  VAR  ($708K)/(0.2%)  (|ll759K)/(5,4%)  EAC;  I370572K 


SCHED  VAR  (|O49K)/(6.0%)  (|9563K)/(4.4%) 

NARRATI\T; 
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Appendix  A:  NSIA  Survey  (Page  18  of  18) 

Page  9 

SURVEY  QUESTIONNAIRE 
Please  circle  the  most  appropriate  responses. 

1.  What  is  your  level  of  education? 

A.  High  School  Graduate 

B.  Some  College 

C.  4  Year  College  Graduate 

D.  Masters 

E.  Post-Masters 

2.  What  is  your  level  of  familiarity  with  Department  of 
Defense  Contracts? 

A.  Never  worked  on  one. 

B.  Some  experience  working  with  DoD. 

C.  Moderate  knowledge  of  DoD  practices. 

D.  Extensive  familiarity  with  DoD  practices. 

3.  What  is  your  level  of  experience  with  Cost/Schedule 
Control  Systems  Criteria? 

A.  None. 

B.  Some  knowledge  of  C/SCSC. 

C.  Moderate  knowledge. 

D.  Extensive  familiarity  with  C/SCSC. 

4.  What  is  your  level  of  experience  with  the  Cost 
Performance  Report? 

A.  None. 

B.  Some  knowledge  of  the  CPR. 

C.  Moderate  knowledge  of  the  CPR. 

D.  Extensive  knowledge  of  the  CPR. 

5.  What  experience  do  you  have  writing  Variance  Analyses? 

A .  None . 

B.  Some  experience. 

C.  Moderate  experience. 

D.  Extensive  experience. 

6.  What  is  your  level  o'"  familiarity  with  the  Program 
Director’s  Assessment  Report? 

A.  I  don’t  know  what  it  is. 

B.  Some  familiarity  with  the  PDAR. 

C.  I  have  moderate  familiarity  with  the  PDAR. 

D.  I  have  extensive  familiarity  with  the  PDAR. 
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Appendix  B:  C/SCSC  Survey  (Page  1  of  18) 


Cost/Schedule  Control  Systems  Criteria  Survey 
Wright-Patterson  Air  Force  Base 
May  16,  1989 


Subjects:  Students  of  the  Cost/Schedule  Control  Systems 
Criteria  (C/SCSC)  course,  SYS  362-89C,  at  the  Air  Force 
Institute  of  Technology.  Thirty-seven  of  the  students  were 
government  employees  and  three  were  contractor  personnel. 
The  contractor  responses  will  be  included  as  part  of  the 
National  Security  Industrial  Association  Survey  results  for 
purposes  of  the  thesis  analysis.  There  was  a  mix  in  the 
level  of  knowledge  of  C/SCSC  among  the  students.  Most  had 
at  least  a  moderate  level  of  working  with  the  Cost 
Performance  Report  and  a  moderate  level  of  writing  variance 
analysis  reports. 


Statistics:  40  Administered 
40  Returned 
32  Usable 

16  Control  Group  (1  Contractor) 

16  Experimental  Group  (2  Contractors) 


Error:  I  became  aware  of  one  error  in  the  instrument.  It 

involved  the  restating  of  the  Propulsion  work  breakdown 
structure  element  Budget  At  Completion  (BAC)  and  Estimate  At 
Completion  (EAC)  numbers  on  the  Systems  Test  &  Evaluation 
"JHA  Systems  Management  Report.”  This  error  was  briefed  to 
the  students  prior  to  administering  the  experiment.  (Though 
it  was  inconsequential,  this  paragraph  was  added  to  note 
that  the  error  had  been  noticed.) 
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Appendix  B:  C/SCSC  Survey  (Page  3  of  18) 


Page  2 


INTRODUCTION 


Scenario:  Ms.  Lee,  the  JHA  SYSTEMS  Program  Control  Manager 
on  the  DAGGER  F-26  program,  is  away  on  an  emergency. 

Although  you  are  very  busy  as  a  Program  Control  Manager  on 
another  program,  the  F-26  Program  Manager,  Mr.  Jones,  has 
requested  your  help  since  it  appears  Ms.  Lee  will  be 
unavailable  for  a  few  weeks.  Today  is  the  thirteenth 
working  day  after  the  April  accounting  close.  The  CPR  is 
due  to  the  customer  on  the  17th  working  day  following  the 
accounting  close.  In  order  to  meet  your  deadline,  you  must 
finish  the  CPR  today.  The  data  is  in,  but  the  variance 
analysis  is  yet  to  be  completed. 

Information  Available:  Mr.  Phillips,  the  F-26  Assistant 
Program  Control  Manager,  has  brought  you  the  JHA  Systems 
Management  Reports  which  help  explain  the  cost  situation. 
These  reports  are  required  by  Mr.  Jones  for  all  work 
breakdown  structure  elements  which  have  a  cost  variance  in 
excess  of  10*  for  current  month  or  in  excess  of  5*  and 
$250,000  at  cumulative. 

Task:  Your  immediate  task  is  to  use  the  information  in  the 
CPR  and  JHA  Management  Reports  to  write  up  a  variance 
analysis  report  (included  as  page  7  of  this  package)  for  the 
total  contract  cost  variance  only.  Mr.  Phillips  is  handling 
all  the  schedule  variance  analyses  as  well  as  the  level  2 
and  lower  (if  required)  cost  variance  analyses.  The  vari¬ 
ance  analysis  report  you  write  should  meet  with  approval 
from  Mr.  Jones  and  the  government.  You  have  a  meeting  with 
Mr.  Jones  in  15  minutes  to  discuss  the  overall  status  of  the 
F-26  contract  cost  variance.  Good  luck! 


Note:  Format  5  variance  analysis  guidance  states  the 
following  for  total  contract  reporting: 

"Provide  a  summary  analysis,  identifying  significant 
problems  affecting  performance.  Indicate  corrective  actions 
required,  including  Government  action  where  applicable." 
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Appendix  B:  C/SCSC  Survey  (Page  5  of  18) 


JHA  SYSTEMS 

MANAGEMENT  REPORT 


4/34/89 


COST 

□  SCHEDULE 

□  TECHNICAL 

□  CONTRACTS 

TfBS  ELEMENT:  AIRFRAME 


rirPRFMT  /i%\ 


□  CURRENT  PERIOD  >+10% 

S  CURRENT  PERIOD  >-10% 

□  CUMUUTI\T:  >+5%  k  i250K 
S  CUMUUTI^T  >-5%  k  $350K 


MANAGER;  R.  MARSHALL 


riru/zi;) 

f  »•  i 


r  ^  A  •*  «*  A  b 


COST  VAR  (|599K)/(15.9%)  (|8410K)/(9.S%)  EAC;  |153893K 
SCHED  VAR  ($403K)/(10.7%)  ($4223K)/(4.8%) 


NARRATIVE; 

We  are  still  having  difficulty  with  the 
chemical  structure  of  the  composite  material.  The 
problem  surfaces  are  the  vertical  stabilizer  and 
the  trailing  portions  of  the  wings.  The  problem 
is  that  the  exhaust  from  the  engines  is  reacting 
with  the  chemical  composition  of  the  lifting 
surfaces,  forming  a  thin  coating  on  the  surfaces. 
This  residue  is  affecting  the  flight  character¬ 
istics  of  the  lifting  surfaces.  I  have  discussed 
the  problem  with  Mr.  Andersen  and  he  is  looking 
into  possibly  altering  the  exhaust  heat.  We  have 
been  analyzing  the  residue  to  see  how  we  can  alter 
the  chemical  structure  of  the  composite  material 
and  are  close  to  uncovering  the  problem.  I  expect 
resolution  of  this  problem  by  June  with  minimal 
impact  to  the  program. 
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Appendix  B:  C/SCSC  Survey  (Page  6  of  18) 


JHA  SYSTEMS  fA 

M4NAGEMENT  REPORT 

COST 

□  schedule 

□  TECHNICAL 

□  CONTRACTS 

□  CURRENT  PERIOD  >+10% 

E  CURRENT  PERIOD  >-10% 

□  CUMUUTIVE  >+5%  k  |250K 

S  CUMULATI\T  >-5%  k  $250K 

WBS  ELEMENT:  PROPULSION 

1  MANAGER:  C.  ANDERSEN 

CURRENT/(%) 

CUM/(%)  BAC:  I48233K 

COST  VAR  (|116K)/(10.5%) 

(fie02K)/(5.7%)  EAC:  I52210K 

SCHED  VAR  ($lseK)/(ig.l%)  (f20QlK)/(6,5%) 


NARRATFE: 

We  continue  to  have  a  problem  with  the 
chemical  reaction  between  the  exhaust  and  the 
composite  material.  I  have  tried  altering  the 
exhaust  gas  temperature  (EGT),  but  it  has  not 
affected  the  accumulation  of  sediment  on  the 
vertical  stabilizer  and  trailing  portions  of  the 
wings.  The  changes  in  the  EGT  have  caused  my  WBS 
element  to  incur  the  variances  noted  above.  I 
will  continue  with  my  plans  and  remain  optimistic 
that  the  designed  EGT  is  appropriate.  However,  I 
may  have  to  modify  other  areas  of  the  propulsion, 
hence  my  revised  EAC  is  as  previously  submitted  to 
you  and  noted  above.  The  problem  will  be  resolved 
when  the  AIRFRAME  WBS  Manager  solves  the  chemical 
structure  problem  of  the  composite  material.  My 
discussion  with  Mr.  Marshall  signifies  that  the 
problem  should  be  resolved  by  June. 
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4/24^09 


JHA  SYSTEMS 

^L4NAGEMENT  REPORT 


g  COST 

□  SCHEDULE 

□  TECHNICAL 
n  CONTRACTS 


□  CURRENT  PERIOD  >+10% 
i  CURRENT  PERIOD  >-10% 

□  CUMUUTIVE  >+5%  k  |250K 
0  CUMULATIVE  >-5%  k  i250K 


WBS  ELEMENT;  S’i'S  TEST  k  El^Il  MANAGER;  D.  KTOTO 


CURRENT/(%)  CUM/(%) 
COST  VAR  (|154K)/(24.D%)  (|1011K)/(9,7%) 


BAC;  $27223K 
EAC;  $29093K 


SCHED  VAR  ($145K)/(23.1%)  ($2a20K)/(l4.1%) 


NARRATIV'E; 

The  most  recent  test  suggests  that 
either  the  chemical  composition  of  the  lifting 
surfaces  needs  to  change  or  the  propulsion  exhaust 
gas  temperature  ( EGT )  must  change.  Both  WBS 
managers  are  aware  of  the  recurring  problem.  It 
has  set  me  back  as  noted  above.  The  next  systems 
test  is  in  July.  Both  managers  have  assured  me 
that  the  problem  will  be  fixed  by  then  and  that 
the  system  will  pass  the  test.  No  further  impacts 
are  expected,  except  for  the  unfavorable  cost 
variance  at  completion  as  already  recorded  by  the 
Cost  Accounting  System. 
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Appendix  B:  C/SCSC  Survey  (Page  8  of  18) 


JHA  SYSTEMS  ^ 

\^IANCE  ANALYSIS  REPORT 

0  COST  □  CURRENT  PERIOD  >+10% 

□  SCHEDULE  □  CURRENT  PERIOD  >-10% 

□  TECHNICAL  □  CUMULATI^T  >+5%  k  |250K 

□  CONTRACTS  !2!  CUMULATH^E  >-5%  k  $250K 

WBS  ELEMENT;  TOTAL  CONTRACT!  MANAGER;  R,  JONES 

CURRENT/(%)  CUM/(%)  BAC;  f351092K 

COST  VAR  ($7e9K)/(e,2%)  ($11759K)/(5,4%)  EAC:  $g70572K 

SCHED  VAR  (|649K)/(S.8%)  (|9563K)/(4.4%) 

NARRATH^E; 
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Page 

SURVEY  QUESTIONNAIRE 
Please  circle  the  most  appropriate  responses. 

1 .  What  is  your  level  of  education? 

A.  High  School  Graduate. 

B.  Some  College. 

C.  4  Year  College  Graduate. 

D.  Masters. 

E.  Post-Masters. 

2.  What  is  your  level  of  familiarity  with  Department  of 
Defense  Contracts? 

A.  Never  worked  on  one. 

B.  Some  experience  working  with  DoD. 

C.  Moderate  knowledge  of  DoD  practices. 

D.  Extensive  familiarity  with  DoD  practices. 

3.  What  is  your  level  of  experience  with  Cost/Schedule 
Control  Systems  Criteria? 

A.  None. 

B.  Some  knowledge  of  C/SCSC. 

C.  Moderate  knowledge. 

D.  Extensive  familiarity  with  C/SCSC. 

4.  What  is  your  level  of  experience  with  the  Cost 
Performance  Report? 

A.  None. 

B.  Some  knowledge  of  the  CPR. 

C.  Moderate  knowledge  of  the  CPR. 

D.  Extensive  knowledge  of  the  CPR. 

5.  What  experience  do  you  have  writing  Variance  Analyses? 

A.  None. 

B.  Some  experience. 

C.  Moderate  experience. 

D.  Extensive  experience. 

6.  What  is  your  level  of  familiarity  with  the  Program 
Director’s  Assessment  Report? 

A.  I  don't  know  what  it  is. 

B.  Some  familiarity  with  the  PDAR. 

C.  I  have  moderate  familiarity  with  the  PDAR. 

D.  I  have  extensive  familiarity  with  the  PDAR. 
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MAY  17,  1989 


COST/SCHEDULE  CONTROL  SYSTEMS  CRITERIA 
(SYS  362-89C)  SURVEY 


TO  BE  USED  IN 


THE  EFFECT  OF  FEEDBACK  ON  COST  PERFORMANCE  REPORT  QUALITY" 
MASTERS  THESIS  -  AFIT/GCA/LSY/89S-2 
AIR  FORCE  INSTITUTE  OF  TECHNOLOGY 

CAPTAIN  JUAN  H.  AMARAL 


Table  of  Contents 

Item  Page 

Title  Page  and  Table  of  Contents  1 

Introduction  2 

DAGGER  F-26  CPR  Format  1  3 

JHA  SYSTEMS  Management  Reports  4-6 

Program  Director’s  Comments  7 

JHA  SYSTEMS  Variance  Analysis  Report  8 
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Appendix  B:  C/SCSC  Survey  (Page  11  of  18) 

INTRODUCTION 


Page  2 


Scenario:  Ms.  Lee,  the  JHA  SYSTEMS  Program  Control  Manager 
on  the  DAGGER  F-26  program,  is  away  on  an  emergency.  Al¬ 
though  you  are  very  busy  as  a  Program  Control  Manager  on 
another  program,  the  F-26  Program  Manager,  Mr.  Jones,  has 
requested  your  help  since  it  appears  Ms.  Lee  will  be  un¬ 
available  for  a  few  weeks.  Today  is  the  thirteenth  working 
day  after  the  April  accounting  close.  The  CPR  is  due  to  the 
customer  on  the  17th  working  day  following  the  accounting 
close.  In  order  to  meet  your  deadline,  you  must  finish  the 
CPR  today.  The  data  is  in,  but  the  variance  analysis  is  yet 
to  be  completed. 

Information  Available:  Mr.  Phillips,  the  F-26  Assistant 
Program  Control  Manager,  has  brought  you  the  JHA  Systems 
Management  Reports  which  help  explain  the  cost  situation. 
These  reports  are  required  by  Mr.  Jones  for  all  work  break¬ 
down  structure  elements  which  have  a  cost  variance  in  excess 
of  10%  for  current  month  or  in  e  .cess  of  5%  and  $250,000  at 
cumulative.  Additionally,  Mr.  Phillips  has  brought  you  a 
copy  of  the  customer’s  report,  based  on  last  month’s  data, 
which  the  customer  sends  to  its  headquarters.  Ms.  Lee 
requested  a  copy  of  the  customer  report  to  understand  what 
the  government  does  with  the  data  it  receives  in  the  CPR. 

The  feedback  Ms.  Lee  has  received  from  the  government 
centers  on  the  lack  of  a  complete  explanation  of  the  problem 
areas.  Capt  Hoyt,  the  customer  F-26  program  control 
analyst,  thought  that  sending  Ms.  Lee  a  copy  of  the  previous 
Program  Director’s  Comments  would  show  her  just  how  much 
detail  was  actually  missing  in  the  variance  analysis.  He 
included  handwritten  comments  on  the  copy  he  sent  Ms.  Lee  to 
point  out  that  the  information  the  government  requests  "up 
the  chain"  was  not  making  its  way  into  the  variance  analysis 
reports,  especially  at  total  contract  level. 

Task:  Your  immediate  task  is  to  use  the  information  in  the 

CPR  and  JHA  Management  Reports  to  write  up  a  variance 
analysis  report  (included  as  page  7  of  this  package)  for  the 
total  contract  cost  variance  only.  Mr.  Phillips  is  handling 
all  the  schedule  variance  analyses  as  well  as  the  level  2 
and  lower  (if  required)  cost  variance  analyses.  The  vari¬ 
ance  analysis  report  you  write  should  meet  with  approval 
from  Mr.  Jones  and  the  government.  You  have  a  meeting  with 
Mr.  Jones  in  15  minutes  to  discuss  the  overall  status  of  the 
F-26  contract  cost  variance.  Good  luck! 

Note:  Format  5  variance  analysis  guidance  states  the 

following  for  total  contract  reporting:  "Provide  a  summary 
analysis,  identifying  significant  problems  affecting 
performance.  In  ;ate  corrective  actions  required, 
including  Governi.ient  action  where  applicable.” 
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Appendix  B:  C/SCSC  Survey  (Page  13  of  18) 


JHA  SYSTEMS 

MANAGEMENT  REPORT 


Ur: 
4/24/89 


E  COST 

□  SCHEDULE 

□  'TECHNICAL 

□  CONTRACTS 


WBS  ELEMENT:  AIRFRAME 


rirRRFWT  /l^\ 


□  CURRENT  PERIOD  >+10% 

M  CURRENT  PERIOD  >-10% 

□  CUMULATI^T:  >+5%  k  I250K 
Si  CUMULATIVE  >-5%  k  $250K 


MANAGER:  R.  NL4RSHALL 


now  //^\ 


Pin-  .finiJififlir 

%'  W  ^  ^  V  «*• 


COST  VAR  (|599K)/(15.9%)  ($8416K)/(9,6%)  EAC:  $153093K 

SCHED  VAR  ($403K)/(10.7%)  ($4223K)/(4.8%) 


NARRATIV'E: 

We  are  still  having  difficulty  with  the 
chemical  structure  of  the  composite  material.  The 
problem  surfaces  are  the  vertical  stabilizer  and 
the  trailing  portions  of  the  wings.  The  problem 
is  that  the  exhaust  from  the  engines  is  reacting 
with  the  chemical  composition  of  the  lifting 
surfaces,  forming  a  thin  coating  on  the  surfaces. 
This  residue  is  affecting  the  flight  character¬ 
istics  of  the  lifting  surfaces.  I  have  discussed 
the  problem  with  Mr.  Andersen  and  he  is  looking 
into  possibly  altering  the  exhaust  heat.  We  have 
been  analyzing  the  residue  to  see  how  we  can  alter 
the  chemical  structure  of  the  composite  material 
and  are  close  to  uncovering  the  problem.  I  expect 
resolution  of  this  problem  by  June  with  minimal 
impact  to  the  program. 
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Appendix  B:  C/SCSC  Survey  {Page  14  of  18) 


JHA  SYSTEMS 

M4NAGEMENT  REPORT 


i  COST 

□  CURRENT  PERIOD  >+10% 

□  SCHEDULE 

S  CURRENT  PERIOD  >-10% 

□  TECHNICAL 

□  CUMUUTIV^E  >+5%  &  $250K 

□  CONTRACTS 

0  CUMUUTIVE  >-5%  k  $250K 

TOS  ELEMENT;  PROPULSION 

1  MANAGER:  C.  ANDERSEN 

pirppii’M'r  \ 

CUM/(%}  BAC:  I49233K 

COST  VAR  (|11QK)/(10.5%) 

(|l802K)/(5.7%)  EAC;  S52210K 

SCHED  VAR  (|igQK)/(l5.1%) 

($2061K)/(6.5%) 

NARRATIVE; 

We  continue  to  have  a  problem  with  the 
chemical  reaction  between  the  exhaust  and  the 
composite  material.  I  have  tried  altering  the 
exhaust  gas  temperature  (EGT),  but  it  has  not 
affected  the  accumulation  of  sediment  on  the 
vertical  stabilizer  and  trailing  portions  of  the 
wings.  The  changes  in  the  EGT  have  caused  my  WBS 
element  to  incur  the  variances  noted  above.  I 
will  continue  with  my  plans  and  remain  optimistic 
that  the  designed  EGT  is  appropriate.  However,  I 
may  have  to  modify  other  areas  of  the  propulsion, 
hence  my  revised  EAC  is  as  previously  submitted  to 
you  and  noted  above.  The  problem  will  be  resolved 
when  the  AIRFRAME  WBS  Manager  solves  the  chemical 
structure  problem  of  the  composite  material.  My 
discussion  with  Mr.  Marshall  signifies  that  the 
problem  should  be  resolved  by  June. 
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AS  OF; 
4/24/89 


MANAGEMENT  REPOET 


§  COST 

□  CURRENT  PERIOD  >+10% 

□  SCHEDULE 

S  CURRENT  PERIOD  >-10% 

□  TECHNICAL 

□  CUMUL4TI\T  >+5%  k  $250K 

□  CONTRACTS 

0  CUMULATI\T  >-5%  k  •$250K 

ITBS  ELEMENT:  SYS  TEST  & 

E4^Ali  MANAGER;  D,  KYOTO 

V  V  1  VI  1  X  ,  \  1*  / 

CUM/(%)  BAC;  $27223K 

COST  VAR  ($154K)/(24.n%) 

($1011K)/(9.7%)  EAC;  I29893K 

SCHED  VAR  ($145K)/(23.1%) 

($2020K)/(14.1%) 

NiRRATIV’E: 

The  most  recent  test  suggests  that 
either  the  chemical  composition  of  the  lifting 
surfaces  needs  to  change  or  the  propulsion  exhaust 
gas  temperature  ( EGT )  must  change.  Both  WBS 
managers  are  aware  of  the  recurring  problem.  It 
has  set  me  back  as  noted  above.  The  next  systems 
test  is  in  July.  Both  managers  have  assured  me 
that  the  problem  will  be  fixed  by  then  and  that 
the  system  will  oass  the  test.  No  further  impacts 
are  expected,  except  for  the  unfavorable  cost 
variance  at  completion  as  already  recorded  by  the 
Cost  Accounting  System. 
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♦♦UNCLASSIFIED** 


Page  7 


DAGGER  F-26  May,  1989 

Format  11  -  Program  Director’s  Comments  PREPARED:  04/27/89 

Explanation  of  Y  in  Cost  Performance 


-PROBLEM 

JHA  ?‘'stems  is  experiencing  problems  in  the  airframe.  The 
spec.ric  problem  is  unknown,  but  it  seems  to  be  centered  on 
the  shaping  of  the  composite  materials.  JHA  also  has 
problems  with  the  Propulsion  WBS  element.  The  specific 
problem  is  unknown.  Finally,  JHA  is  falling  behind  in 
System  Test  and  Evaluation. 

-IMPACT: 

There  is  no  indication  that  JHA  will  be  able  to  recover  the 
cost  variance  satisfactorily.  JHA  has  not  specifically 
identified  the  problem  and  does  not  really  know  the  impact 
of  the  problem.  JHA  estimates  a  cost  overrun  of  $7.3 
million,  but  our  estimate  is  a  $15.6  million  overrun. 

-ACTION  TAKEN: 

The  AFPRO  and  DCAS  personnel  have  been  asked  to  look  into 
the  airframe  problem  to  help  determine  the  extent  of  the 
design  shortcomings.  The  program  director  will  meet  the  JHA 
program  manager  to  discuss  the  cost  situation. 

-GET  WELL:^’*' 

With  AFPRO,  DCAS,  and  JHA  management  attention,  trend 
reversal  expected  Jun  89. 

^  Ml  L<<  ’•  y  .  rc^*") 

.  X  «.wviu.r«.  -^vS 

varvAiM:<.  S  9^ 

pUavc  ct**itwiote  ^  Uvcl  oJH  {o*»«.r 

IcJtVs  . 

-p  V^P<  tkii  WlfS'. 
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Appendix  B:  C/SCSC  Survey  (Page  17  of  18) 


JHA 

VAEIANCE 

SYSTEMS  4/’l^a9 

ANALYSIS  REPORT 

0  COST 

□  CURRENT  PERIOD  >+10% 

□  SCHEDULE 

□  CURRENT  PERIOD  >-10% 

□  TECHNICAL 

□  CUMUUTIVE  >+5%  k  |250K 

□  CONTRACTS 

13  CUMUUTIVE  >-5%  6;  ^250K 
-  ■'  -1'  --  ■■  ■—  -- —  1 

WB5  ELEMENT:  TOTAL  CONTRACTj  MiLN^4GER:  R.  JONES 


CURRENT/(%)  CUM/(%)  BAC:  $3&1S93K 

COST  VAR  (1798K)/(a.2%)  (lll759K)/(5,4%)  EAC;  $370572K 
SCHED  VAR  (|64aK)/(6.9%)  (|9563K)/(4.4?T.) 

NARRATIVE; 
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Page  9 


SURVEY  QUESTIONNAIRE 
Please  circle  the  most  appropriate  responses. 

1 .  What  is  your  level  of  education? 

A.  High  School  Graduate 

B.  Some  College 

C.  4  Year  College  Graduate 

D.  Masters 

E.  Post-Masters 

2.  What  is  your  level  of  familiarity  with  Department  of 
Defense  Contracts? 

A.  Never  worked  on  one. 

B.  Some  experience  working  with  DoD. 

C.  Moderate  knowledge  of  DoD  practices. 

D.  Extensive  familiarity  with  DoD  practices. 

3.  What  is  your  level  of  experience  with  Cost/Schedule 
Control  Systems  Criteria? 

A .  None . 

B.  Some  knowledge  of  C/SCSC. 

C.  Moderate  knowledge. 

D.  Extensive  familiarity  with  C/SCSC. 

4.  What  is  your  level  of  experience  with  the  Cost 
Performance  Report? 

A.  None . 

B.  Some  knowledge  of  the  CPR. 

C.  Moderate  knowledge  of  the  CPR. 

D.  Extensive  knowledge  of  the  CPR. 

5.  What  experience  do  you  have  writing  Variance  Analyses? 

A.  None. 

B.  Some  experience. 

C.  Moderatf  experience. 

D.  Extensi experience. 

6.  What  is  your  level  of  familiarity  with  the  Program 
Director’s  Assessment  Report? 

A.  I  don’t  know  what  it  is. 

B.  Some  familiarity  with  the  PDAR. 

C.  I  have  noderate  familiarity  with  the  PDAR. 

D.  I  have  extensive  familiarity  with  the  PDAR. 


82 


Appendix  C:  Rater  Calibration  Package  (Page  1  of  3) 


LSG  (Capt  Juan  Amaral,  AV  255-4437)  12  May  89 

Expert  Evaluation  of  Thesis  Results 

To:  Lt  Col  Bowman,  ASD/ACCM,  Wri ght-Patterson  AFB  OH  45433 

Major  Cohen,  SSD/ACCI ,  Los  Angeles  AFB  CA  90009 
Mr  Gardella,  ESD/ACCI,  Hanscom  AFB  MA  01731 

1.  As  Capt  Amaral  discussed  with  you,  attached  is  the  first 
phase  of  the  expert  evaluation  of  results  obtained  from  the 
two  experiments  he  conducted.  During  phase  one,  he  needs  to 
calibrate  the  judges  (you  are  one  of  them)  for  his  experi¬ 
ments.  The  package  should  be  self-explanatory.  Please 
return  attachment  three  in  return  envelope. 

2.  Upon  completion  of  this  phase,  Capt  Amaral  will  send  you 
a  package  of  instructions  for  phase  two  of  the  evaluation 
process.  During  phase  two  you  will  be  rating  the  actual 
responses  he  received  as  a  result  of  administering  the 
experiment  to  two  groups,  contractor  personnel  at  a  National 
Security  Industrial  Association  Meeting  and  government  per¬ 
sonnel  at  an  AFIT  Professional  Continuing  Education  Course. 

3.  Thank  you  for  your  support  of  this  academic  endeavor. 
jlteMcO  'fi'i 

ANTHONY  H.  PRESUTTI,  Lt  Col,  USAF  3  Atch 

Head,  Dept  of  Quantitative  1.  Background  Information 

Management  2.  Copy  of  Survey 

School  of  Systems  and  Logistics  3.  Response  Evaluation 
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Appendix  C:  Rater  Calibration  Package  (Page  2  of  3) 

Atch  1 


Evaluation  of  Thesis  Survey  Results 
Phase  I  -  Rater  Calibration 


INTRODUCTION:  During  the  first  phase  of  the  survey  results 

evaluation,  the  raters  are  to  rank-order  the  following  sam¬ 
ple  variance  analyses  (Attachment  3)  according  to  their  qua¬ 
lity.  This  initial  rating  will  be  compared  to  the  other  two 
raters’  evaluations  to  establish  if  a  consistent  assessment 
of  variance  analysis  quality  exists  among  expert  raters. 

SCENARIO:  The  attached  survey  (Atch  2)  has  been  adminis¬ 
tered  to  approximately  45  contractor  and  45  government  per¬ 
sonnel.  As  stated  on  page  2  of  the  survey,  the  task  in¬ 
volved  the  preparation  of  a  variance  analysis  report  for 
Level  1  of  the  fictional  JHA  Systems  Dagger  F-26  contract. 
Page  3,  the  Dagger  F-26  Cost  Performance  Report,  was  pro¬ 
vided  to  the  participants  as  back-up  information.  Each 
participant  was  verbally  instructed  to  concentrate  on  the 
information  provided  on  the  JHA  Systems  Management  Reports 
(pages  4-6).  These  three  reports  are  the  only  ones  required 
by  JHA  management  and  provide  information  on  the  three  ele¬ 
ments  exceeding  reportable  thresholds.  As  you  might  expect, 
they  form  the  basis  for  the  CPR  Format  5  JHA  provides  to  the 
government.  Each  participant  wrote  his  or  her  answer  on 
page  7  of  the  survey.  Page  8  was  given  to  obtain  demo¬ 
graphic  information  on  the  participants. 

EVALUATION:  Attachment  3  contains  three  sample  responses  to 
the  task  described  above.  Each  response  would  have  been 
written  on  page  7  of  the  survey  and  would  have  supported  the 
financial  data  found  on  that  page.  Please  evaluate  these 
three  variance  analyses  and  return  them  in  the  enclosed 
envelope.  Each  response  has  a  blank  line  towards  the  bottom 
where  you  are  to  place  either  a  1,  2,  or  3.  For  the  vari¬ 
ance  analysis  you  find  as  the  best,  place  a  "1"  on  the  line. 
The  second  best  variance  analysis  should  be  rated  "2"  on  the 
line.  Finally,  place  a  “3'’  on  the  line  corresponding  to  the 
remaining  variance  analysis.  Please  do  not  assign  any  other 
ratings  to  the  variance  analyses. 

CRITERIA  FOR  EVALUATION:  In  evaluating  the  three  variance 
analyses,  please  consider  their  quality  on  the  basis  of 
general  guidelines  for  variance  analysis.  Additionally,  and 
most  important  for  this  thesis,  keep  in  mind  the  limited 
information  presented.  The  guiding  principle  in  assessing 
the  quality  of  each  variance  should  be,  "How  good  is  this 
variance  analysis,  given  the  limited  information?” 
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Appendix  C:  Rater  Calibration  Package  (Page  3  of  3) 

Atch  3 


Sample  Survey  Responses 
Sample  Response  »1: 

The  overall  contract  variance  deals  with  minor  design  and 
implementation  problems  in  the  contract.  Because  of  these 
setbacks,  we  are  not  meeting  our  schedule  and  cost  budgets. 
The  financial  impact  of  this  variance  is  noted  above.  We 
have  personne’  from  every  level  of  the  company  working  on 
these  minor  problems  and  we  expect  them  to  be  resolved  soon, 
with  no  impact  to  the  overall  contract. 

Seunple  Response  #1  Rating:  3 


Sample  Response  »2: 

The  overall  contract  variance  is  a  direct  result  of 
residue  forming  on  the  surfaces  of  the  wings  and  similar 
areas.  Since  we  did  not  anticipate  these  problems,  our 
schedule  position  is  worse  than  expected  (BCWS  >  BCWP)  and 
our  cost  position  is  also  worse  than  expected  (ACWP  >  BCWP). 
The  exact  values  are  noted  above.  The  responsible  WBS 
managers  are  handling  their  individual  areas  and  there  does 
not  seem  to  be  any  further  problem  anticipated.  The  remain¬ 
ing  schedule  milestones  will  be  met  with  no  impact  to  the 
overall  contract. 


Sample  Response  #2  Rating: _ Z. 


Sample  Response  #3: 

The  overall  contract  variance  is  driven  by  three  key 
areas:  Airframe,  Propulsion,  and  Systems  Test  and  Evalua¬ 
tion.  The  main  problem  is  that  exhaust  gases  are  causing  a 
chemical  reaction  with  the  composite  material  on  the  lifting 
surfaces  of  the  aircraft.  Changes  to  the  Exhaust  Gas  Tem¬ 
perature  (EGT)  have  not  resolved  the  problem.  Although 
these  changes  have  resulted  in  increased  costs  in  the  Pro¬ 
pulsion  and  ST&E  WBS  elements,  they  have  provided  clues  to 
the  anomalies  in  the  chemical  structure  of  the  composite 
material.  The  Airframe  WBS  manager  is  close  to  resolving 
the  composite  material  problem  and  expects  to  have  it  solved 
by  June.  The  current  and  at  completion  variances  are  noted 
above.  There  is  no  impact  expected  on  the  July  Systems  Test 
and  no  impact  to  the  overall  contract. 

Sample  Response  #3  Rating:  1 
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Appendix  D:  Response  Evaluation  Package  (Page  1  of  37 ) 


LSG  (Capt  Juan  Amaral,  AV  255-4437)  12  Jun  89 

Expert  Evaluation  of  Thesis  Results 

To:  Lt  Col  Bowman,  ASD/ACCM 

Major  Cohen,  SSD/ACCI 
Mr  Gardella,  ESD/ACCI 

1.  Thank  you  for  your  prompt  response  to  the  first  phase  of 
the  thesis  results  rating  process.  The  results  of  the  rater 
calibration  suggest  that  there  is  a  consistent  rating  pro¬ 
cess  between  the  three  raters. 

2.  As  Capt  Amaral  has  discussed  with  you,  attached  you  will 
find  the  results  he  obtained  from  administering  his  experi¬ 
ment  to  two  different  groups.  The  second  phase  of  the 
thesis  results  evaluation  involves  the  rating  of  these 
actual  results. 

3.  Attachment  1  is  a  set  of  instructions  concerning  the 
evaluation  process  and  provides  you  with  guidance  on  how  to 
rate  the  responses.  Attachment  2,  enclosed  for  your  infor¬ 
mation,  is  a  copy  of  the  instrument  Capt  Amaral  administered 
to  the  subjects.  Attachment  3  contains  one-third  of  the 
usable  results  he  obtained.  Please  evaluate  each  response 
and  return  Attachment  3  in  the  enclosed  envelope. 

4.  Once  again,  thank  you  for  your  support  of  this  academic 
endeavor . 

ANTHONY  H.  PRESUTTI ,  Lt  Col,  USAF  3  Atch 

Head,  Dept  of  Quantitative  1.  Background  Information 

Management  2.  Copy  of  Survey 

School  of  Systems  and  Logistics  3.  Response  Evaluation 
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Appendix  D:  Response  Evaluation  Package  (Page  2  of  37) 


Atch  1  (1  of  2) 

Evaluation  of  Thesis  Survey  Results 
Phase  II  -  Response  Evaluation 

INTRODUCTION:  During  the  first  phase,  the  expert  raters 

were  given  three  sample  responses  to  rank-order.  The  re¬ 
sponses  received  from  the  three  raters  indicate  that  there 
is,  in  fact,  consistent  rating  among  the  three  raters. 

During  the  second  phase  of  the  survey  results  evaluation, 
the  raters  are  to  provide  a  sco^e  for  each  of  the  actual 
responses  received  during  the  two  experiments.  Each  rater 
will  be  randomly  assigned  one-third  of  the  total  usable 
responses  accumulated  during  the  experiments. 

SCENARIO:  The  attached  survey  (Atch  2)  has  been  adminis¬ 

tered  to  46  contractor  and  40  government  personnel .  As 
stated  on  page  2  of  the  survey,  the  task  involved  the  pre¬ 
paration  of  a  variance  analysis  report  for  Level  1  of  the 
fictional  JHA  Systems  Dagger  F-26  contract.  Page  3,  the 
Dagger  F-26  Cost  Performance  Report,  was  provided  to  the 
participants  as  back-up  information.  Each  participant  was 
verbally  instructed  to  concentrate  on  the  information  pro¬ 
vided  on  the  JHA  Systems  Management  Reports  (pages  4-6). 
These  three  reports  are  the  only  ones  currently  required  by 
JHA  management  and  provide  information  on  the  three  elements 
exceeding  reportable  thresholds.  As  you  might  expect,  they 
form  the  basis  for  the  CPR  Format  5  JHA  provides  to  the 
government. 

Each  participant  was  randomly  given  either  an  8-page  or 
9-page  version  of  the  instrument.  Attachment  2  is  a  copy  of 
the  8-page  non-feedback  version.  The  feedback  version 
contains  an  additional  page  which  provides  feedback  to  the 
subjects.  To  maintain  the  integrity  of  the  response  evalua¬ 
tions,  the  raters  will  receive  a  copy  of  the  feedback  ver¬ 
sion  upon  rating  the  responses.  It  is  possible  that  provid¬ 
ing  a  copy  of  the  feedback  version  to  the  raters  prior  to 
their  rating  of  the  responses  may  bias  the  evaluation  pro¬ 
cess  . 

EVALUATION:  A  Likert  Scale  appears  at  the  top  of  each  page 
of  Attachment  3.  Please  evaluate  each  of  the  responses  in 
accordance  with  the  information  shown  at  the  top  of  each 
page.  Each  response  would  have  supported  the  financial  data 
found  on  the  appropriate  response  page  of  the  experiment. 
Each  response  has  a  blank  line  towards  the  bottom  where  you 
are  to  place  your  evaluation.  Please  keep  the  evaluations 
in  whole  numbers;  in  other  words,  only  the  values  1,  2,  3, 

4,  5,  6,  or  7  may  appear  as  evaluations.  Please  do  not 
assign  any  other  ratings  to  the  variance  analyses. 
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Appendix  D:  Response  Evaluation  Package  (Page  3  of  37) 


Atch  1  (2  of  2) 


CRITERIA  FOR  EVALUATION:  In  evaluating  the  responses, 
please  consider  their  quality  on  the  basis  of  general  guide¬ 
lines  for  variance  analysis.  Additionally,  and  most  impor¬ 
tant  for  this  thesis,  consider  the  limited  information 
presented.  The  guiding  principle  in  assessing  the  quality 
of  each  variance  should  be,  "How  good  is  this  variance 
analysis,  given  the  limited  information?" 
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Appendix  D:  Response  Evaluation  Package  (Page  4  of  37) 


Given  the  limited  information  provided  and  general  vari¬ 
ance  analysis  guidelines,  each  of  the  following  responses: 


1 

2 

3 

4 

5 

6 
7 


Contains 

Contains 

value. 

Contains 

Contains 

Contains 

Contains 

Contai ns 

tion . 


no  variance  analysis  value, 
a  very  limited  amount  of  variance  analysis 

a  limited  amount  of  variance  analysis  value, 
some  variance  analysis  value, 
a  moderate  amount  of  variance  analysis  value, 
a  large  amount  of  variance  analysis  value, 
a  variance  analysis  of  all  available  informa- 


001 


The  drivers  for  the  above  cost  variance  are  primarily  in 
the  Airframe,  Propulsion,  and  System  Test  and  Evaluation  WBS 
elements.  The  cause  for  these  variances  is  a  problem  with 
the  chemical  structure  of  the  composite  material  on  the 
airframe  vertical  stabilizer  and  trailing  portions  of  the 
wings.  Engine  exhaust  is  causing  a  reaction  with  the  chemi¬ 
cal  which  forms  a  residue  on  the  surfaces  affecting  the 
flight  characteristics  of  the  surfaces. 

Possible  solutions  such  as  changing  the  chemical  struc¬ 
ture  of  the  material  and/or  changing  the  temperature  of  the 
exhaust  are  being  examined. 

We  expect  the  problem  will  be  solved  by  June.  This 
will  be  prior  to  the  next  System  Test  and  Evaluation. 

Impact  to  the  program  to  be  minimal. 


Response  Rating _ ± 


002 

The  negative  cost  variance  situation  is  the  result  of 
negative  reaction  of  the  chemical  structure  of  the  composite 
material  used  in  the  vertical  stabilizer  and  trailing  por¬ 
tions  of  the  wings  and  the  exhaust  from  the  engine.  This 
results  in  a  buildup  of  residue  on  the  stabilizer  and  wings 
which  affects  the  flight  characteristics  of  the  aircraft. 
Additional  costs  were  incurred  because  of  retest  and  ex¬ 
perimentation.  The  corrective  action  is  to  modify  the 
chemical  composition  of  the  material  used  on  the  wings  and 
vertical  stabilizers.  This  will  result  in  additional  cost 
to  the  overall  program. 

Response  Rating  3 
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Appendix  D:  Response  Evaluation  Package  (Page  5  of  37) 


Given  the  limited  information  provided  and  general  vari¬ 
ance  analysis  guidelines,  each  of  the  following  responses: 


1 

2 

3 

4 

5 

6 
7 


Contai ns 

Contains 

value. 

Contains 

Contai ns 

Contains 

Contains 

Contains 

tion . 


no  variance  analysis  value, 
a  very  limited  amount  of  variance  analysis 

a  limited  amount  of  variance  analysis  value, 
some  variance  analysis  value, 
a  moderate  amount  of  variance  analysis  value, 
a  large  amount  of  variance  analysis  value, 
a  variance  analysis  of  all  available  informa- 


003 

The  5.4%  cost  variance  as  noted  above  is  a  result  of 
problems  encountered  in  the  airframe  and  propulsion  systems. 
Either  the  composite  material  will  be  restructured,  or  the 
exhaust  heat  will  be  changed  to  correct  the  problem.  Sys¬ 
tems  Test  will  then  be  conducted  in  July  to  finalize  the 
results.  The  projected  EAC  appears  in  light  of  the  problems 
encountered  based  on  a  CPI  of  .85  in  the  airframe  and  pro¬ 
pulsion  and  a  TCPI  of  .90  on  the  airframe  and  propulsion 
systems. 


Response  Rating _ 


004 

The  current  unfavorable  cost  and  schedule  variance  are 
related  to  problems  dealing  with  the  chemical  reaction 
between  the  exhaust  and  the  composite  material  (WBS  3  Airfr¬ 
ame  and  WBS  3  Propulsion).  These  problems  have  adversely 
impacted  System  Test  and  Evaluation.  It  is  anticipated  that 
management  reserve  will  be  used  to  provide  the  required 
funds  to  meet  the  above  problems. 


Response  Rating _ I 


Appendix  D:  Response  Evaluation  Package  (Page  6  of  37) 


Given  the  limited  information  provided  and  general  vari¬ 
ance  analysis  guidelines,  each  of  the  following  responses: 


1 

2 

3 

4 

5 

6 
7 


Contai ns 
Contai ns 
value. 
Contai ns 
Contai ns 
Contai ns 
Contains 
Contains 
tion . 


no  variance  analysis  value, 
a  very  limited  amount  of  variance  analysis 

a  limited  amount  of  variance  analysis  value, 
some  variance  analysis  value, 
a  moderate  amount  of  variance  analysis  value, 
a  large  amount  of  variance  analysis  value, 
a  variance  analysis  of  all  available  informa- 


005 

The  <11,759K>  cost  variance  is  due  to  WBS  elements  as 
fol lows: 

-  Airframe  <8,416K> 

-  Propulsion  <1,802K> 

-  Systems  Test  &  Eval  <1,811K> 

The  principle  causes  of  this  variance  are  due  to  diffi¬ 
culty  with  the  chemical  structure  of  the  composite  material 
(in  the  airframe  WBS).  The  exhaust  from  the  engines  is 
reacting  with  the  chemical  composition  of  the  lifting  sur¬ 
faces,  forming  a  coating.  This  residue  is  affecting  the 
flight  characteristics  of  the  lifting  surfaces.  This  ac¬ 
counts  for  74*  of  the  total  variance.  The  propulsion  WBS 
has  an  impact  from  the  chemical  reaction  between  the  exhaust 
and  the  composite  material.  This  contributes  20*  of  the 
variance.  Systems  Test  and  Evaluation  is  impacted  due  to 
failures  incurred  as  a  result  of  the  above  mentioned  air¬ 
frame  and  propulsion  WBS  problems. 

Impact:  A  one  month  slip  is  projected  to  resolve  airframe 

problem.  Expected  completion  date  is  June.  Once  airframe 
is  resolved,  propulsion  will  regain  schedule.  No  impact  to 
test  is  projected  pending  adequate  resolution  of  airframe 
and  propulsion  activity.  Expected  completion  date  for  next 
Systems  Test  is  July. 

Corrective  Action:  Analysis  of  residue  to  alter  chemical 
structure  of  the  composite  material.  Coordinate  with  pro¬ 
pulsion  to  incorporate  design  changes  and  monitor  Systems 
Test  and  Evaluation. 

Response  Rating  6 
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Appendix  D:  Response  Evaluation  Package  (Page  7  of  37) 


Given  the  limited  information  provided  and  general  vari¬ 
ance  analysis  guidelines,  each  of  the  following  responses: 


1 

2 

3 

4 

5 

6 
7 


Contains 

Contains 

value. 

Contains 

Contains 

Contains 

Contains 

Contains 

tion. 


no  variance  analysis  value, 
a  very  limited  amount  of  variance  analysis 

a  limited  amount  of  variance  analysis  value, 
some  variance  analysis  value, 
a  moderate  amount  of  variance  analysis  value, 
a  large  amount  of  variance  analysis  value, 
a  variance  analysis  of  all  available  informa- 


006 


There  does  not  seem  to  be  sufficient  communication  bet¬ 
ween  the  parties  to  establish  true  cause  of  the  problem, 
therefore,  it  may  be  expected  to  continue.  On  this  basis, 
the  overrun  will  grow.  There  is  no  indication  that  problem 
will  be  resolved  in  the  near  term. 


Response  Rating _ 1 


007 


The  air  vehicle  is  encountering  problems  in  the  area  of 
the  flight  characteristics  of  the  lifting  surfaces.  This  is 
due  to  the  surface  of  the  vertical  stabilizer  and  trailing 
portions  of  the  wings  accumulating  sediment. 


Corrective  action:  Investigations  are  underway  to  deter¬ 
mine  if  the  problems  are  created  by  the  chemical  structure 
of  the  composite  material  or  the  exhaust  gas  temperature  of 
the  propulsion  subsystem.  Once  determined,  testing  will 
continue  as  planned  in  July. 


Impact:  Costs  as  reflected  in  the  EAC  should  be  recog¬ 


nized  as  reported, 
anticipated.  This 
ted  in  the  current 
scheduled . 


but  no  additional  impacts  to  the  EAC  are 
problem  will  be  fixed  in  June  as  reflec- 
EAC  and  testing  will  be  continued  as 


Response  Rating _ S. 
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Appendix  D: 


(Page  8  of  37) 


Response  Evaluation  Package 

Given  the  limited  information  provided  and  general  vari¬ 
ance  analysis  guidelines,  each  of  the  following  responses: 


1 

2 

3 

4 

5 

6 
7 


Contains 
Contai ns 
value. 
Contains 
Contains 
Contains 
Contai ns 
Contains 
tion . 


no  variance  analysis  value, 
a  very  limited  amount  of  variance  analysis 

a  limited  amount  of  variance  analysis  value, 
some  variance  analysis  value, 
a  moderate  amount  of  variance  analysis  value 
a  large  amount  of  variance  analysis  value, 
a  variance  analysis  of  all  available  informa 


008 

Problem:  Chemical  reaction  between  the  exhaust  and 
composite  material.  Unfavorable  variance  due  to  resolving 
this  issue. 


Impact:  Currently  project  variance  stated  above  based 

on  current  data. 


Action:  WBS  element  managers  are  coordinating  resolu¬ 
tion  and  expect  to  resolve  the  issue  by  June. 

Response  Rating  2 


009 

The  causes  of  the  variances  have  been  identified.  These 
are  the  problems  associated  with  the  engine  exhaust  and  the 
composition  of  the  lifting  surface  material.  The  exhaust 
reacts  with  the  material  to  produce  an  undesirable  condition 
that  affects  performance.  This  accounts  for  the  System  Test 
and  Evaluation  variance  also.  The  managers  of  the  Airframe 
(R.  Marshall)  and  Propulsion  (C.  Andersen)  WBS  elements  are 
aware  of  the  problem  and  are  taking  corrective  actions  to 
correct  the  problem.  The  managers  assure  that  the  problem 
will  be  corrected  by  adjusting  the  material  of  the  lifting 
surface  or  the  exhaust  temperature  of  the  engine.  This 
corrective  action  should  be  completed  by  June.  No  further 
impacts  are  expected  except  for  the  unfavorable  cost  vari¬ 
ance  at  completion  as  already  recorded  by  the  Cost  Account¬ 
ing  System. 


Response  Rating _ 2. 
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Appendix  D:  Response  Evaluation  Package  (Page  9  of  37) 


Given  the  limited  information  provided  and  general  vari¬ 
ance  analysis  guidelines,  each  of  the  following  responses: 

1  -  Contains  no  variance  analysis  value. 

2  -  Contains  a  very  limited  amount  of  variance  analysis 

val ue . 

3  -  Contains  a  limited  amount  of  variance  analysis  value. 

4  -  Contains  some  variance  analysis  value. 

5  -  Contains  a  moderate  amount  of  variance  analysis  value. 

6  -  Contains  a  large  amount  of  variance  analysis  value. 

7  -  Contains  a  variance  analysis  of  all  available  informa 

tion. 


010 

The  major  contribution  to  the  current  and  cumulative  cost 
variance  is  the  difficulties  with  the  chemical  structure  of 
the  composite  material.  Resolution  of  the  problem  is  ex¬ 
pected  by  June,  with  the  next  system  test  in  July.  Resolu¬ 
tion  of  this  problem  will  impact  cost  performance  of  the 
three  major  contributors  to  the  current  cost  overrun. 

Impact  to  the  program  at  this  point  is  expected  to  be  mini¬ 
mal.  Government  will  monitor  both  cost  and  schedule  perfor¬ 
mance  closely  through  the  expected  corrective  action  date. 

Response  Rating  3 


01  1 

The  program  is  scheduled  for  Systems  Test  in  May  and  the 
current  outlook  is  optimistic.  The  cost  estimate  at  comple¬ 
tion  assumes  June  resolution  of  an  exhaust  sediment  problem. 
Special  attention  will  be  put  in  place  to  maximize  the  April 
solution  of  the  exhaust  sediment  problem  and  a  July  Systems 
Test.  The  program  EAC  will  be  reassessed  if  the  June  or 
July  checkpoints  become  unachievable. 

Response  Rating  3 
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Appendix  D:  Response  Evaluation  Package  (Page  10  of  37) 


Given  the  limited  information  provided  and  general  vari¬ 
ance  analysis  guidelines,  each  of  the  following  responses: 


1 

2 

3 

4 

5 

6 
7 


Contains 
Contains 
value. 
Contai ns 
Contai ns 
Contai ns 
Contains 
Contains 
tion . 


no  variance  analysis  value, 
a  very  limited  amount  of  variance  analysis 

a  limited  amount  of  variance  analysis  value, 
some  variance  analysis  value, 
a  moderate  amount  of  variance  analysis  value, 
a  large  amount  of  variance  analysis  value, 
a  variance  analysis  of  all  available  informa- 


012 

Cum  cost  variance  continues  to  be  caused  by  design  prob¬ 
lems  involving  the  vertical  stabilizer  and  the  exhaust  heat 
impact  on  it.  Major  redesign  of  engine  placement  is  being 
evaluated  and  reported.  EAC  and  projected  VAC  may  change 
drastically  following  the  design  meeting  later  this  month. 

Response  Rating  3 


013 

We  continue  to  have  a  problem  with  the  chemical  reaction 
between  the  exhaust  from  the  propulsion  units  and  the  com¬ 
posite  materials  of  the  vertical  stabilizer  and  wing  trail¬ 
ing  edges.  The  EGT  has  been  adjusted  without  any  improve¬ 
ment  shown  during  testing.  The  makeup  of  the  composite  is 
being  altered  and  tested  to  find  a  non-oxidizing  formula. 

The  airframe  design  team  is  confident  of  a  quick  resolution, 
but  a  high  degree  of  risk  is  involved.  Each  month  of  slip¬ 
page  is  costing  us  $5,000  and  our  management  reserve  will 
not  cover  current  and  potential  overruns.  The  current 
$7,000  overrun  will  go  as  high  as  $20,000. 

Response  Rating  4 


014 

The  total  cost  variance  of  5.4%  exceeds  a  desireable 
variance.  The  current  period  percentage  of  8.2%  is  increas¬ 
ing  from  the  cumulative  5.4%.  If  the  problem  is  cleared  up 
by  June,  we  will  have  a  much  more  favorable  position  at 
completion. 


Response  Rating _ Z 


95 


Appendix  D:  Response  Evaluation  Package  (Page  11  of  37) 


Given  the  limited  information  provided  and  general  vari¬ 
ance  analysis  guidelines,  each  of  the  following  responses: 


1 

2 

3 

4 

5 

6 
7 


Contai ns 

Contains 

value. 

Contains 

Contai ns 

Contains 

Contains 

Contains 

ti  on . 


no  variance  analysis  value, 
a  very  limited  amount  of  variance  analysis 

a  limited  amount  of  variance  analysis  value, 
some  variance  analysis  value, 
a  moderate  amount  of  variance  analysis  value, 
a  large  amount  of  variance  analysis  value, 
a  variance  analysis  of  all  available  informa- 


015 

There  is  an  unresolved  issue  between  the  airframe  and  the 
propulsion  systems  segments.  We  are  establishing  an  in¬ 
dependent  group  to  study  the  evidence  and  propose  a  correc¬ 
tive  action  for  whether  it  is  EGT  or  composite  related. 

Final  impact  is  unknown  at  this  time  and  will  be  reported, 
with  the  corrective  action  plan,  next  reporting  period. 


Response  Rating _ 2. 


016 

There  is  a  problem  with  a  chemical  reaction  in  the  com¬ 
posite  material  structure.  The  corrective  action  centers 
around  either  changing  the  exhaust  gas  temperature  or  revis¬ 
ing  the  chemical  composition  of  the  composite  material.  We 
have  a  plan  to  identify  the  correct  chemical  composition  of 
the  composite  material  and  the  specification  for  EGT  prior 
to  the  scheduled  systems  test  in  July. 

Current  cost  overruns  appear  to  be  unrecoverable. 
However,  the  impact  on  the  EAC  should  not  reflect  a  severe 
deterioration  beyond  that  experienced  cum  to  date. 

Response  Rating  3 
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Appendix  D:  Response  Evaluation  Package  (Page  12  of  37) 


Given  the  limited  information  provided  and  general  vari¬ 
ance  analysis  guidelines,  each  of  the  following  responses: 


1 

2 

3 

4 

5 

6 
7 


Contai ns 

Contains 

value. 

Contains 

Contains 

Contains 

Contains 

Contains 

tion . 


no  variance  analysis  value, 
a  very  limited  amount  of  variance  analysis 

a  limited  amount  of  variance  analysis  value, 
some  variance  analysis  value, 
a  moderate  amount  of  variance  analysis  value, 
a  large  amount  of  variance  analysis  value, 
a  variance  analysis  of  all  available  informa- 


017 

The  project  major  cost  drivers  are  the  difficulties  with 
the  chemical  structure  of  the  composite  material.  The 
problem  may  be  alleviated  by  changes  to  the  EGT.  Up-to-date 
changes  have  not  materialized  and  further  studies  are  war¬ 
ranted  in  the  chemical  composite  material.  Projected  solu¬ 
tions  to  either  change  in  chemical  nature  of  material  and 
EGT  are  expected  in  June.  At  that  time,  the  reported  cost 
variance  could  be  stabilized.  If  no  solution  is  reached  by 
June,  other  solutions  may  be  necessary. 


Response  Rating _ 2, 


018 

The  total  contract  at  completion  cost  variance  is  due  to 
a  combination  of  problems  in  the  airframe  and  propulsion 
areas.  The  problems  will  be  resolved,  but  there  is  no 
indication  that  the  overrun  to  date  will  be  compensated  for 
with  future  efficiencies. 

The  program  will  be  back  on  schedule  as  of  July  test¬ 
ing,  assuming  no  further  technical  problems  arise.  This  EAC 
variance  reflects  only  technical  problems  to  date,  and  does 
not  assume  additional  technical  problems. 

Response  Rating  3 
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Appendix  D:  Response  Evaluation  Package  (Page  13  of  37) 


Given  the  limited  information  provided  and  general  vari¬ 
ance  analysis  guidelines,  each  of  the  following  responses: 


1 

2 

3 

4 

5 

6 
7 


Contains 

Contains 

value. 

Contains 

Contains 

Contains 

Contains 

Contains 

tion . 


no  variance  analysis  value, 
a  very  limited  amount  of  variance  analysis 

a  limited  amount  of  variance  analysis  value, 
some  variance  analysis  value, 
a  moderate  amount  of  variance  analysis  value, 
a  large  amount  of  variance  analysis  value, 
a  variance  analysis  of  all  available  informa- 


019 

This  project  is  behind  schedule  and  overrunning  cost  due 
to  unexpected  difficulties  with  the  chemical  structure  of 
the  composite  material  on  the  vertical  stabilizer  and  the 
trailing  portions  of  the  wing.  To  date  we  are  analyzing  the 
residue  forming  on  these  surfaces  (caused  by  the  engine 
exhaust)  to  determine  whether  if  it  will  be  possible  to 
alter  the  chemical  composition  of  the  composite  material  to 
improve  flight  characteristics.  This  analysis  will  be 
completed  by  June.  Management  reserve  will  be  applied  to 
cost  overrun  and  the  addition  of  personnel  to  make  up  for 
schedule  slippage. 


Response  Rating _ 2, 


020 

We  are  continuing  to  have  the  problem  in  the  area  of  the 
chemical  composition  of  the  composite  material  on  the  air¬ 
craft.  Exhaust  from  the  engines  creates  a  thin  coating  on 
the  surfaces,  which  negatively  affects  aircraft  performance. 
The  anticipated  solutions  involve  either  redirecting  exhaust 
gas  or  changing  the  chemical  composition.  The  cost  varian¬ 
ces  experienced  are  related  to  the  corrective  action  inves¬ 
tigations  and  testing  we  have  encountered  to  remedy  this 
situation.  Corrective  action  will  take  place  in  July,  at 
which  time  we  expect  to  regain  performance  with  no  current 
period  CV,  but  not  being  able  to  recover  the  cumulative  CV. 
This  is  reflected  in  the  revised  EAC. 


Response  Rating _ 2 
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Appendix  D:  Response  Evaluation  Package  (Page  14  of  37) 


Given  the  limited  information  provided  and  general  vari¬ 
ance  analysis  guidelines,  each  of  the  following  responses: 


1 

2 

3 

4 

5 

6 
7 


Contai ns 
Contains 
value. 
Contains 
Contains 
Contains 
Contai ns 
Contai ns 
tion . 


no  variance  analysis  value, 
a  very  limited  amount  of  variance  analysis 

a  limited  amount  of  variance  analysis  value, 
some  variance  analysis  value, 
a  moderate  amount  of  variance  analysis  value. 
2  large  amount  of  variance  analysis  value, 
a  variance  analysis  of  all  available  informa- 


021 

Problem  Definition:  Chemical  residue  buildup  on  airframe 
control  surfaces  due  to  reaction  between  composite  material 
and  exhaust  gas  is  causing  altered  flight  characteristics 
and  product  retest. 

Corrective  Action:  Propulsion  is  studying  ways  of 
mitigating  exhaust  action  by  varying  exhaust  gas  tempera¬ 
ture.  Structures  is  reviewing  the  composite  material  for  a 
possible  change  in  chemical  structure.  Results  are  expected 
by  June  -  in  time  for  the  July  retest.  Progress  indicates 
the  June  date  will  be  met. 

Program  Impact:  Unrecoverable  cost  overrun. 

Response  Rating  3 


022 

The  total  program  cost  variance  results  from  problems 
with  the  airframe  activities  and  propulsion  activities.  The 
resolution  these  problems  are  ongoing  and  are  expected  to 
be  resolved  by  June.  There  is  an  anticipated  cost  impact  at 
completion  of  $7,260K. 


Response  Rating _ ± 
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Appendix  D:  Response  Evaluation  Package  (Page  15  of  37) 


Given  the  limited  information  provided  and  general  vari¬ 
ance  analysis  guidelines,  each  of  the  following  responses: 

1  -  Contains  no  variance  analysis  value. 

2  -  Contains  a  very  limited  amount  of  variance  analysis 

value. 

3  -  Contains  a  limited  amount  of  variance  analysis  value. 

4  -  Contains  some  variance  analysis  value. 

5  -  Contains  a  moderate  amount  of  variance  analysis  value. 

6  -  Contains  a  large  amount  of  variance  analysis  value. 

7  -  Contains  a  variance  analysis  of  all  available  informa¬ 

tion. 


023 

There  is  a  significant  cost  variance  for  June  in  the 
Propulsion/Airframe  areas:  The  issue  is  also  affecting  test. 

Technical  Background:  Integration  issue  revolving  around 
wing  trailing  edge  and  propulsion  exhaust  interaction  af¬ 
fecting  stability  and  control  of  the  air  vehicle.  Changes 
to  propulsion  do  not  appear  feasible.  Modification  of  wing 
trailing  edge  structure  composition  appears  to  be  best  or 
quickest  fix.  Fix  will  be  defined  by  June.  Testing  com¬ 
plete  by  July. 

Cost  Analysis:  Because  of  the  nature  of  this  problem,  it 
does  not  appear  that  we  will  be  able  to  recover  this  vari¬ 
ance.  Contractor’s  assessment  of  damage  limits  variance  to 
7.3M$.  This  assumes  no  problems  in  fix  incorporation  or  in 
testing.  Our  assessment  is  that  this  problem  will  require 
an  alternative  approach,  with  a  total  variance  anticipated 
of  15M$. 


Response  Rating _ 5, 


024 

The  contract  level  variances 


consist  of  (in  $K)  Schedule  Cost 


Cur 

Cum 

Cur 

Cum 

1 . 

Airframe  (Level  13) 

599 

4,023 

599 

8,416 

2. 

Propulsion  (Level  3) 

166 

2,061 

116 

1 ,802 

3. 

Test  &  Evaluation  (Level 

12)  145 

2,620 

154 

1,811 

We  are  having  problems  with  the  chemical  structure  in 
the  airframe.  In  Propulsion,  a  problem  with  chemical  reac¬ 
tion  between  exhaust  and  composite  material.  These  two  are 
causing  a  problem  in  Systems  Test  and  Evaluation. 

Response  Rating  2 
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Appendix  D:  Response  Evaluation  Package  (Page  16  of  37) 


Given  the  limited  information  provided  and  general  vari¬ 
ance  analysis  guidelines,  each  of  the  following  responses: 


1 

2 

3 

4 

5 

6 
7 


Contains 

Contains 

value. 

Contains 

Contains 

Contains 

Contai ns 

Contai ns 

tion. 


no  variance  analysis  value, 
a  very  limited  amount  of  variance  analysis 

a  limited  amount  of  variance  analysis  value, 
some  variance  analysis  value, 
a  moderate  amount  of  variance  analysis  value, 
a  large  amount  of  variance  analysis  value, 
a  variance  analysis  of  all  available  informa- 


025 

Problem  Definition:  Test  failure  of  the  performance  on 
the  F-26  airframe. 

Problem  Cause:  Exhaust  gas  temperature  is  deteriorat¬ 
ing  the  lift  performance  of  the  wings  and  vertical  stabil- 
i  zer . 

Corrective  Action:  Test  to  be  rerun  after  both  composi¬ 
tion  of  the  airframe  (wings  and  vertical  stabilizer)  has 
been  changed  and  possible  lowering  of  EGT.  Test  to  be  rerun 
in  July. 

Program  Impact:  Additional  costs  on  composition  change 
is  expected.  Additionally,  costs  for  retesting  have  driven 
the  program  EAC  to  $370,572K.  No  further  costs  are  expected 
if  composition  change  meets  test  specifications. 

Response  Rating  4 


026 

We  are  experiencing  residue  collection  on  the  lifting 
surfaces  due  to  the  exhaust  from  the  engines.  We  are  cur¬ 
rently  analyzing  the  residue  to  possibly  alter  the  chemical 
structure  of  the  composite  material.  We  are  close  to  re¬ 
solving  the  problem.  Get  well  date  in  June. 

Response  Rating _ 3_ 
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Appendix  D;  Response  Evaluation  Package  (Page  17  of  37) 


Given  the  limited  information  provided  and  general  vari¬ 
ance  analysis  guidelines,  each  of  the  following  responses: 

1  -  Contains  no  variance  analysis  value. 

2  -  Contains  a  very  limited  amount  of  variance  analysis 

value. 

3  -  Contains  a  limited  amount  of  variance  analysis  value. 

4  -  Contains  some  variance  analysis  value. 

5  -  Contains  a  moderate  amount  of  variance  analysis  value. 

6  -  Contains  a  large  amount  of  variance  analysis  value. 

7  -  Contains  a  variance  analysis  of  all  available  informa¬ 

tion. 


027 

The  Negative  Cost  Variance  is  driven  by  the  Air  Vehicle, 
Systems  Test  and  Evaluation,  and  Propulsion.  This  negative 
variance  is  attributed  to  chemical  reactions  with  material 
on  the  vertical  stabilizer  and  trailing  portions  of  the 
wings.  Solution  by  changing  exhaust  temperature  or  the 
composition  of  lifter  surfaces.  This  problem  should  be 
corrected  by  June  with  minimal  impact.  The  APPRO  and  DCAS 
are  both  involved  in  the  effort  to  correct  this  problem  in  a 
timely  manner. 


Response  Rating _ 3. 


028 

A  problem  with  the  exhaust  gas  residue  buildup  on  lift 
surfaces  continues  to  plague  completion  of  design  change  and 
flight  test.  A  several -pronged  attack  has  been  mounted  to 
determine  if  1)  either  exhaust  gas  chemical  or  temperature 
changes  can  be  affected,  2)  exhausted  gases  can  be  deflected 
and  3)  composite  part  chemical  composition  can  be  changed. 

We  are  making  good  headway  in  the  chemical  composition  area. 
We  expect  to  expend  an  additional  $6  million  to  correct  this 
problem  and  have  the  fix  in  place  by  the  end  of  June. 

Response  Rating  2 


029 

We  continue  to  have  a  chemical  problem  in  the  airframe 
area,  but  anticipate  resolution  of  the  problem  by  July.  The 
latest  EAC  reflects  the  associated  cost  growth,  but  we  be¬ 
lieve  the  technical  problems  are  containable  within  the 
stated  EAC.  The  cost  growth  displayed  in  the  Propulsion  and 
Systems  Test  areas  will  also  be  contained  within  the  stated 
EAC. 

Response  Rating  2 
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Appendix  D:  Response  Evaluation  Package  (Page  18  of  37) 


Given  the  limited  information  provided  and  general  vari¬ 
ance  analysis  guidelines,  each  of  the  following  responses: 


1 

2 

3 

4 

5 

6 
7 


Contains 
Contains 
value. 
Contains 
Contains 
Contai ns 
Contains 
Contai ns 
tion . 


no  variance  analysis  value, 
a  very  limited  amount  of  variance  analysis 

a  limited  amount  of  variance  analysis  value, 
some  variance  analysis  value, 
a  moderate  amount  of  variance  analysis  value, 
a  large  amount  of  variance  analysis  value, 
a  variance  analysis  of  all  available  informa- 


030 

Interface  problems  between  the  Airframe  and  Propulsion 
subsystems  are  causing  cost  growth  in  both  areas.  The 
problem  centers  around  interactions  between  the  Propulsion 
system  exhaust  gases  reacting  with  and  collecting  on  air¬ 
frame  surfaces  (specifically,  vertical  stabilizer  and  trail¬ 
ing  portions  of  the  wings).  Both  aspects  of  the  problem  are 
being  worked  aggressively  with  resolution  expected  by  July. 
Furthermore,  Systems  Test  and  Evaluation  is  being  adversely 
impacted  due  to  the  uncertainties  indicated  above. 


Response  Rating _ §. 


031 


could 

ted. 


Test  results  to  date  clearly  identify  a  potential  ma- 
'ial’s  problem  which,  if  not  corrected  in  a  timely  manner, 
lead  to  larger  investment  in  coatings  than  anticipa- 


Please  refer  to  Airframe,  Propulsion,  and  Systems  Test, 
and  Evaluation  VARs  for  cause,  impact,  and  current  cor¬ 
rective  action  plans.  We  would  encourage  technical  coor¬ 


dination  with  Air  Force  labs  should 
materials  data  that  would  be  useful 


the  SPO  have 
in  resolving 


access  to 
this  issue. 


Response  Rating _ 3_ 
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Appendix  D:  Response  Evaluation  Package  (Page  19  of  37) 


Given  ^he  limited  information  provided  and  general  vari¬ 
ance  analysis  guidelines,  each  of  the  following  responses: 


1 

2 

3 

4 

5 

6 
7 


Contains 

Contains 

value. 

Contains 

Contains 

Contains 

Contains 

Contains 

tion. 


no  variance  analysis  value, 
a  very  limited  amount  of  variance  analysis 

a  limited  amount  of  variance  analysis  value, 
some  variance  analysis  value, 
a  moderate  amount  of  variance  analysis  value, 
a  large  amount  of  variance  analysis  value, 
a  variance  analysis  of  all  available  informa- 


032 


The  chemical  structure  of  the  composite  material  is 
causing  a  chemical  reaction  between  the  exhaust  and  the 
composite  material.  The  problem  surfaces  in  the  vertical 
stabilizer  and  the  trailing  portions  of  the  wings.  Our 
corrective  action  plan  was  to  alter  the  exhaust  gas  tempera¬ 
ture,  but  it  has  not  affected  the  accumulation  of  the  sedi¬ 
ment  on  the  vertical  stabilizer  and  trailing  portions  of  the 
wings. 

Response  Rating  2 

033 


Difficulties  have  been  experienced 
action  between  exhaust  gas  and  certai 
Management  is  closely  monitoring  the 
tests  are  scheduled  for  July,  we  anti 
terioration  in  cost  other  than  those 
date . 


with  the  chemical  re- 
n  composite  materials, 
situation.  Though 
cipate  no  further  de- 
experienced  cum  to 


Response  Rating _ Z, 


034 


Significant  Problems:  1)  Chemical  interaction  between 
propulsion  and  airframe  composites  have  required  additional 
study  and  analysis,  2)  the  problem  has  to  be  resolved  prior 
to  Systems  Test  in  July,  and,  3)  it  has  resulted  in  the 
stated  cost  growth  and  approximately  1  month  schedule  slip 
for  completing  Systems  Test  and  Evaluation. 


Corrective  Action: 
posite,  and,  2)  alter 


1)  Change  chemical  structure  of 
exhaust  heat  profile. 


com- 


Response  Rating _ 5, 
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Appendix  D:  Response  Evaluation  Package  (Page  20  of  37) 


Given  the  limited  information  provided  and  genera)  vari¬ 
ance  analysis  guidelines,  each  of  the  following  responses: 


1 

2 

3 

4 

5 

6 
7 


Contains 

Contains 

value. 

Contains 

Contai ns 

Contains 

Contains 

Contains 

tion. 


no  variance  analysis  value, 
a  very  limited  amount  of  variance  analysis 

a  limited  amount  of  variance  analysis  value, 
some  variance  analysis  value, 
a  moderate  amount  of  variance  analysis  value, 
a  large  amount  of  variance  analysis  value, 
a  variance  analysis  of  all  available  informa- 


035 

The  current  cost  variance  and  cumulative  cost  variance 
are  due  to  problems  with  the  interaction  of  the  Exhaust  Gas 
Temperatures  and  the  composite  materials  they  are  passing 
over.  Solutions  are  being  tried  and  it  is  expected  the 
problems  will  be  resolved  without  further  impact.  No  gov¬ 
ernment  action  required. 


Response  Rating _ Z. 


036 

According  to  Mr.  Marshall,  a  resolution  to  the  composite 
material  problem  will  be  found  by  June  with  minimal  impact 
to  the  program.  Mr.  Anderson  is  supposed  to  be  working  this 
problem  by  altering  the  EGT.  However,  this  is  causing  other 
problems  which  in  turn  is  causing  T  4  E  problems. 

At  this  point,  there  seems  to  be  no  way  this  problem 
can  be  resolved  by  June  without  additional  impacts  to  the 
program  in  other  areas.  I  feel  the  cost  and  schedule  vari¬ 
ances  will  continue  to  break  the  threshold  by  an  even  bigger 
variance. 


Response  Rating _ 2. 
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Appendix  D:  Response  Evaluation  Package  (Page  21  of  37) 


Given  the  limited  information  provided  and  general  vari¬ 
ance  analysis  guidelines,  each  of  the  following  responses: 


1 

2 

3 

4 

5 

6 
7 


Contains 

Contains 

value. 

Contains 

Contains 

Contains 

Contains 

Contains 

tion. 


no  variance  analysis  value, 
a  very  limited  amount  of  variance  analysis 

a  limited  amount  of  variance  analysis  value, 
some  variance  analysis  value, 
a  moderate  amount  of  variance  analysis  value, 
a  large  amount  of  variance  analysis  value, 
a  variance  analysis  of  all  available  informa- 


037 

The  cumulative  cost  variance  of  $1,759  is  primarily  dri¬ 
ven  by  technical  chemical  reaction  problems  associated  with 
WBS  elements  2.3,  Airframe,  2.4,  Propulsion,  and  System  Test 
and  Evaluation.  The  technical  problem  is  associated  with 
either  a  change  to  the  chemical  composition  of  the  lifting 
surfaces,  or  a  change  to  the  propulsion  gas  temperature. 

The  EAC  on  the  program  is  impacted  because  a  change  to 
the  airframe  or  the  propulsion  must  be  made.  The  total  EAC 
of  $370,572K  will  increase  by  an  amount  to  be  determined 
when  the  technical  solution  is  identified. 

The  problem  is  being  worked  under  the  direction  of  the 
program  manager  and  the  managers  for  Airframe  and  Propul¬ 
sion. 


Response  Rating _ 2, 


038 

Analysis:  Cumulative  cost  variances  is  the  result  of  a 
problem  with  the  composite  material  used  in  the  vertical 
stabilizer  and  trailing  portion  of  the  wings.  The  exhaust 
reacts  with  this  material,  which  results  in  a  buildup  of 
residue  which  impacts  the  flight  characteristics. 

Corrective  Action:  Current  plans  are  to  alter  the 
composite  material  which  will  be  unaffected  by  the  exhaust. 
This  will  be  corrected  by  June. 

Program  Impact:  Cost  impact  noted  thus  far  will  re¬ 
main.  If,  however,  this  fix  does  not  solve  the  problem, 
additional  cost  will  be  incurred  on  the  EAC. 

Response  Rating _ 2_ 
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Appendix  D:  Response  Evaluation  Package  (Page  22  of  37) 


Given  the  limited  information  provided  and  general  vari¬ 
ance  analysis  guidelines,  each  of  the  following  responses: 


1 

2 

3 

4 

5 

6 
7 


Contains 
Contai ns 
value. 
Contains 
Contains 
Contai ns 
Contains 
Contai ns 
tion . 


no  variance  analysis  value, 
a  very  limited  amount  of  variance  analysis 

a  limited  amount  of  variance  analysis  value, 
some  variance  analysis  value, 
a  moderate  amount  of  variance  analysis  value, 
a  large  amount  of  variance  analysis  value, 
a  variance  analysis  of  all  available  informa- 


039 

Chemical  structure  of  composite  material  continues  to 
present  problems  in  achieving  the  contract  cost  and  perfor¬ 
mance  goals.  We  are  working  the  concerns  as  a  team,  but  it 
appears  some  modifications  are  necessary  to  accomplish  our 
required  targets.  This  has  been  a  recurring  problem  and  we 
have  been  working  on  our  significant  concerns.  In  order  to 
correct  the  problem  and  bring  the  program  back  on  line,  I  am 
going  to  revisit  the  requirements  and  recommend  a  possible 
replanning  of  the  budget. 


Response  Rating _ Z. 


040 

Both  the  cost  and  schedule  variances  are  caused  by  air¬ 
frame  and  propulsion  problems.  Of  the  total  cost  variance, 
75%  is  airframe  related  and  deals  with  chemical  structure  of 
the  composite  material  reacting  with  depositing  residue  on 
the  lifting  structures.  We  are  working  the  problem  in 
changing  the  chemical  structure  and  also  modifying  the 
position  of  the  airframe  surfaces. 

Both  the  managers  responsible  for  correction  of  the 
anomalies  have  established  corrective  action  plans  which 
will  be  complete  month  end  July. 

The  EAC  has  been  updated  to  accommodate  these  problems. 

Response  Rating  3 
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Appendix  D:  Response  Evaluation  Package  (Page  23  of  37) 


Given  the  limited  information  provided  and  general  vari¬ 
ance  analysis  guidelines,  each  of  the  following  responses: 


1 

2 

3 

4 

5 

6 
7 


Contai ns 

Contains 

val ue . 

Contains 

Contains 

Contains 

Contains 

Contains 

tion. 


no  variance  analysis  value, 
a  very  limited  amount  of  variance  analysis 

a  limited  amount  of  variance  analysis  value, 
some  variance  analysis  value, 
a  moderate  amount  of  variance  analysis  value, 
a  large  amount  of  variance  analysis  value, 
a  variance  analysis  of  all  available  informa- 


041 

Total  contract  cost  variance  of  7,260K  is  largely  due  to 
the  airframe  and  propulsion  areas.  Due  to  these  problem 
areas  and  tests  to  be  conducted  in  June,  no  make-up  in  cost 
is  expected  soon.  Total  contract  will  overrun  by  at  least 
7,260. 


Response  Rating _ 1 


042 

This  project  is  currently  experiencing  problems  which 
impact  both  the  airframe  and  the  propulsion  WBS  element 
significantly.  There  is  a  chemical  reaction  between  the 
engine’s  exhaust  gases  and  the  composite  material  used  in 
the  wing  trailing  portions  and  the  vertical  stabilizer. 

Both  element  managers  are  showing  variances  due  to  the 
additional  testing  that  each  have  done. 

We  feel  that  the  most  feasible  change  will  be  to  the 
chemical  makeup  of  the  composite  elements,  and  we  are  close 
to  isolating  the  problem.  We  feel  that  the  problem  will  be 
solved  by  June  with  minimal  program  impact. 

Response  Rating  4 
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Appendix  D:  Response  Evaluation  Package  (Page  24  of  37) 


Given  the  limited  information  provided  and  general  vari¬ 
ance  analysis  guidelines,  each  of  the  following  responses: 


1 

2 

3 

4 

5 

6 
7 


Contains 
Contains 
val ue . 
Contains 
Contai ns 
Contains 
Contai ns 
Contains 
tion , 


no  variance  analysis  value, 
a  very  limited  amount  of  variance  analysis 

a  limited  amount  of  variance  analysis  value, 
some  variance  analysis  value, 
a  moderate  amount  of  variance  analysis  value, 
a  large  amount  of  variance  analysis  value, 
a  variance  analysis  of  all  available  informa- 


043 


The  cost  and  schedule  variance  for  the  current  period 
reflected  in  the  airframe,  propulsion,  and  system  test  and 
evaluation,  WBS  elements,  are  related.  The  variances  are 
caused  by  problems  we  have  encountered  due  to  a  reaction 
between  the  composite  surface  material  and  the  exhaust  gas 
from  the  engine.  We  are  working  to  find  a  solution,  and 
estimate  establishing  a  successful  work  around  plan  by  June. 
The  government  should  monitor  progress  including  reviewing 
the  proposed  work  around  plan  we  will  submit  for  feasi¬ 
bility,  and  evaluate  the  reliability  of  the  June  estimate  we 
will  provide. 


Response  Rating _ 3. 


044 

As  M/E  4/89  the  F26  contract  has  exceeded  the  cost  vari¬ 
ance  as  noticed  above.  The  cost  drivers  of  this  variance 
are  the  WBS  elements  airframe,  propulsion,  and  System  Test 
and  Evaluation.  Within  the  airframe  element,  there  seems  to 
be  some  concern  with  the  chemical  structure  of  the  composite 
material  on  the  lifting  surfaces  reacting  with  the  exhaust 
gas..  One  solution  to  this  is  the  possibility  of  altering 
the  exhaust  heat  (EGt)  in  WBS  propulsion.  However,  because 
of  these  changes  in  the  propulsion  element,  the  costs  are 
exceeding  the  thresholds  and  causing  variance.  These  2 
managers  are  in  accord  with  each  other  and  anticipate  a 
solution  by  June.  In  addition  to  the  WBS  elements,  Systems 
Test  and  Evaluation  is  subsequently  exceeding  its  thresh¬ 
olds.  All  problems/concerns  will  be  resolved  by  June,  1 
month  prior  to  Systems  and  Evaluation.  I  anticipate  no 
further  variance  at  completion! 


Response  Rating _ 3, 
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Appendix  0: 


(Page  25  of  37) 


Given  the  limited  information  provided  and  general  vari¬ 
ance  analysis  guidelines,  each  of  the  following  responses: 


Contains 
Contains 
val ue. 
Contains 
Contains 
Contains 
Contai ns 
Contai ns 
t  i  on . 


no  variance  analysis  value, 
a  very  limited  amount  of  variance  analysis 

a  limited  amount  of  variance  analysis  value, 
some  variance  analysis  value, 
a  moderate  amount  of  variance  analysis  value, 
a  large  amount  of  variance  analysis  value, 
a  variance  analysis  of  all  available  informa- 


The  major  problem  areas  in  the  total  program  cost  are 
Test  and  Evaluation,  Propulsion,  and  Airframe.  These  are 
related  problems  in  that  there  is  a  chemical  reaction  be¬ 
tween  the  exhaust  and  composite  material  affecting  the 
vertical  stabilizer  and  trailing  portion  of  the  wings. 

The  problem  is  being  worked  by  running  additional  tests 
and  analyzing  the  residue  to  see  how  the  chemical  structure 
can  be  altered.  A  solution  is  expected  in  June,  with  mini¬ 
mal  impact  to  the  program. 

Response  Rating  2 


The  significant  problem  resulting  in  the  current  cost 
overrun  is  the  effect  of  the  gas  exhaust  on  the  composite 
materials  of  the  wings  and  vertical  stabilizer.  Corrective 
actions  are  being  formulated  in  the  airframe  organization 
which  is  reviewing  composite  material  alternatives.  The 
propulsion  group  is  reviewing  alternatives  to  reduce  gas 
residues  on  the  composites.  A  determination  of  the  best 
alternative  should  be  accomplished  by  June.  At  this  time, 
there  is  no  government  action  requested. 

Response  Rating  3 


The  problem  of  the  engine  exhaust  reaction  with  the 
composite  material  has  been  investigated  but  clear  respon¬ 
sibility  has  not  been  set.  Mr.  Anderson  may  notify  other 
areas  of  propulsion. 

Response  Rating _ ; 


no 


Appendix  D: 


(Page  26  of  37) 


Given  the  limited  information  provided  and  general  vari¬ 
ance  analysis  guidelines,  each  of  the  following  responses: 


1  -  Contains  no  variance  analysis  value. 

2  -  Contains  a  very  limited  amount  of  variance  analysis 

value . 

3  -  Contains  a  limited  amount  of  variance  analysis  value. 

4  -  Contains  some  variance  analysis  value. 

5  -  Contains  a  moderate  amount  of  variance  analysis  value. 

6  -  Contains  a  large  amount  of  variance  analysis  value. 

7  -  Contains  a  variance  analysis  of  all  available  informa¬ 

tion. 


048 


Problem:  The  major  problem  encountered  is  that  the 
exhaust  from  the  engines  is  causing  sediment  buildup  affect¬ 
ing  flight  performance.  Based  on  the  EGT,  chemical  reac¬ 
tions  are  occurring  with  the  composite  material  on  the 
vertical  stabilizer  and  trailing  wing  sections.  Added  costs 
associated  with  varying  the  EGT  and  unanticipated  testing 
have  resulted  in  the  majority  of  the  overrun. 

Impact:  Cost  impact  is  reflected  in  the  EAC  above. 

Action  Taken:  We  have  evaluated  the  chemical  composi¬ 
tion  of  the  lifting  surfaces  and  varied  the  EGT  to  resolve 
the  problem  and  reduce  the  impact. 

Get  Well:  Problem  should  be  resolved  by  June,  and 
thus,  will  not  impact  the  next  test. 

Response  Rating  4 

049 

Main  problem  is  due  to  need  for  redesign  of  airframe 
composite  materials  due  to  reaction  of  current  composite 
structure  with  engine  heat.  New  composite  to  be  selected  by 
end  of  June,  with  minimal  cost  impact. 

The  problem  of  airframe  structure  reacting  to  engine 
heat  considered  lowering  EGT  as  an  alternative,  causing 
current  period  CV  for  propulsion.  This  alternative  was 
rejected  when  redesign  of  composites  for  airframe  was  deter¬ 
mined  to  be  the  correct  solution. 

System  Test  and  Evaluation  was  delayed  as  result  of  the 
Airframe  and  Propulsion  problems. 

Response  Rating  3 
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Appendix  D:  Response  Evaluation  Package  (Page  27  of  37) 


Given  the  limited  information  provided  and  general  vari¬ 
ance  analysis  guidelines,  each  of  the  following  responses: 


1 

2 

3 

4 

5 

6 
7 


Contains 

Contains 

value. 

Contains 

Contains 

Contai ns 

Contains 

Contains 

tion. 


no  variance  analysis  value, 
a  very  limited  amount  of  variance  analysis 

a  limited  amount  of  variance  analysis  value, 
some  variance  analysis  value, 
a  moderate  amount  of  variance  analysis  value, 
a  large  amount  of  variance  analysis  value, 
a  variance  analysis  of  all  available  informa- 


050 

JHA  continues  to  experience  a  cost  variance  due  mainly  to 
System  Test  and  Evaluation  and  propulsion  WBS  element. 

Exact  problem  is  unknown,  however,  problem  is  being  assessed 
by  Airframe  Manager  with  indications  that  a  solution  will  be 
available  by  June.  Cost  Variance  continues  to  increase. 


Response  Rating _ 1 


The  problem  being  experienced  is  cause  by  the  chemical 
reaction  between  the  exhaust  and  the  composite  material. 
Once  the  airframe  WBS  Manager  solves  the  structure  problem, 
no  further  impact  is  expected  after  June. 

Response  Rating _ 2_ 


052 

A  chemical  reaction  problem  between  exhaust  gas  and 
composite  material  continues  with  little  success  in  solely 
changing  exhaust  gas  temperature.  Other  areas  of  the  propu¬ 
lsion  system  may  require  modification.  The  chemical  com¬ 
position  of  composite  materials  is  also  being  investigated. 
Problems  should  be  resolved  by  next  Systems  Test  in  July. 

No  further  impacts  are  expected  on  the  at-completion  costs 
as  a  result  of  these  problems. 


Response  Rating _ 2. 
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Appendix  D:  Response  Evaluation  Package  (Page  28  of  37) 


Given  the  limited  information  provided  and  general  vari¬ 
ance  analysis  guidelines,  each  of  the  following  responses: 


1 

2 

3 

4 

5 

6 
7 


Contains 
Contains 
value. 
Contai ns 
Contains 
Contai ns 
Contai ns 
Contains 
tion. 


no  variance  analysis  value, 
a  very  limited  amount  of  variance  analysis 

a  limited  amount  of  variance  analysis  value, 
some  variance  analysis  value, 
a  moderate  amount  of  variance  analysis  value, 
a  large  amount  of  variance  analysis  value, 
a  variance  analysis  of  all  available  informa- 


053 

The  cost  and  schedule  variances  for  period  ending  4/29/89 
level  03  Airframe  and  Propulsion  are  noted  above.  System 
Test  and  Evaluation  should  also  be  considered. 

Detail  analysis  of  the  above  cost  drivers  is  being 
performed  by  the  responsible  managers. 

The  cost  variances  are  within  the  assigned  thresholds 
for  current  month  and  .4%  over  for  cumulative  date. 

Response  Rating  1 


054 

The  major  contributors  to  the  program’s  cost  variance  are 
the  Airframe  and  Propulsion  WBS  elements,  which  in  turn,  are 
affecting  the  Systems  Test  and  Evaluation  WBS  element. 

Response  Rating  1 
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Appendix  D:  Response  Evaluation  Package  (Page  29  of  37) 


Given  the  limited  information  provided  and  general  vari¬ 
ance  analysis  guidelines,  each  of  the  following  responses: 


1 

2 

3 

4 

5 

6 
7 


Contains 

Contains 

val ue. 

Contains 

Contains 

Contains 

Contains 

Contains 

tion . 


no  variance  analysis  value, 
a  very  limited  amount  of  variance  analysis 

a  limited  amount  of  variance  analysis  value, 
some  variance  analysis  value, 
a  moderate  amount  of  variance  analysis  value, 
a  large  amount  of  variance  analysis  value, 
a  variance  analysis  of  all  available  informa- 


055 


-  PROBLEM: 

JHA  Systems  is  experiencing  problems  with  the  interaction 
between  the  engine  exhaust  temperature  and  the  composite 
material  used  in  wing  and  vertical  stabilizer  control  sur¬ 
faces. 


-  IMPACT: 

JHA  is  reviewing  the  problem  two  ways.  We  are  attempting 
to  find  a  less  reactive  chemical  composite  material  for  the 
control  surfaces.  We  are  also  attempting  to  lower  the 
temperature  of  the  exhaust  gases.  Our  estimate  is  that  a 
solution  will  be  found  by  June. 


-  ACTION  TAKEN: 

Responsible  managers  are  locking  into  the  problem  for  fur¬ 
ther  details  to  suggest  a  corrective  action. 

-  GET  WELL: 

Trend  reversal:  June,  1989 


Response  Rating _ ± 


056 

The  total  cost  for  JHA  is  driven  by  Airframe  ($8.4M  to 
date,  $14.4M  at  completion)  and  Propulsion  ($1.8M  to  date, 
$2.7  at  completion).  The  problems  arise  from  the  chemical 
reactions  of  the  exhaust  system  with  the  composite  materials 
of  the  airframe  wings  and  empennage.  No  solution  to  the 
problem  has  been  found  although  tests  are  still  underway. 
These  problems  are  driving  test  delays.  Ineffective  iden¬ 
tification  will  cost  more  than  projected  to  date,  but  fur¬ 
ther  analysis  of  EAC  cannot  be  completed  until  problem  is 
identified. 

Response  Rating  4 
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Appendix  D:  Response  Evaluation  Package  (Page  30  of  37) 


Given  the  limited  information  provided  and  general  vari¬ 
ance  analysis  guidelines,  each  of  the  following  responses: 

1  -  Contains  no  variance  analysis  value. 

2  -  Contains  a  very  limited  amount  of  variance  analysis 

value. 

3  -  Contains  a  limited  amount  of  variance  analysis  value. 

4  -  Contains  some  variance  analysis  value. 

5  -  Contains  a  moderate  amount  of  variance  analysis  value. 

6  -  Contains  a  large  amount  of  variance  analysis  value. 

7  -  Contains  a  variance  analysis  of  all  available  informa¬ 

tion. 


057 

Problem:  Exhaust  from  the  engines  is  reacting  with  the 

chemical  composition  of  the  lifting  surfaces. 

Impact:  The  problem  affects  the  flight  characteristics 
of  the  lifting  surfaces.  Some  cost  growth  has  emerged,  but 
this  is  not  critical. 

Action  Taken:  We  are  trying  to  alter  the  EGT  and/or  to 
change  the  chemical  composition  of  the  composite  material. 

Completion  Date:  June  1989. 


Response  Rating _ 2, 


058 

Continued  problem  of  chemical  reaction  between  exhaust 
and  composite  material  of  lifting  surfaces.  Corrective 
action  is  underway  to  alter  the  chemical  structure  of  the 
composite  material  by  the  end  of  June.  This  problem  has 
resulted  in  increased  cost  as  reflected  in  new  contract  EAC. 

The  effectiveness  of  the  problem  resolution  will  be 
unknown  until  tests  are  conducted  in  July.  There  has  been 
no  provision  made  in  EAC  for  additional  cost  impact  due  to 
test  failure  or  further  corrective  action  on  this  problem. 

Response  Rating  4 
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Appendix  D:  Response  Evaluation  Package  (Page  31  of  37) 


Given  the  limited  information  provided  and  general  vari¬ 
ance  analysis  guidelines,  each  of  the  following  responses: 


1 

2 

3 

4 

5 

6 
7 


Contai ns 

Contains 

val ue . 

Contains 

Contains 

Contains 

Contains 

Contains 

tion. 


no  variance  analysis  value, 
a  very  limited  amount  of  variance  analysis 

a  limited  amount  of  variance  analysis  value, 
some  variance  analysis  value, 
a  moderate  amount  of  variance  analysis  value, 
a  large  amount  of  variance  analysis  value, 
a  variance  analysis  of  all  available  informa- 


059 


In  summary,  the  composite  material  problems  identified  in 
WBS  areas  for  Airframe,  Propulsion  and  System  Test  and 
Evaluation  will  be  resolved  by  contract  end  with  a  $1.1759M 
overrun  representing  a  5.4%  negative  cost  variance.  Exten¬ 
sive  meetings  with  middle  and  upper  management’s  attention 
will  raise  the  visibility  within  the  company.  We  also 
expect  closer  coordination  with  the  government’s  program 
manager  and  PLO  to  ensure  your  visibility  into  solution 
methodology. 


Response  Rating _ 1 


060 

Current  total  contract  cost  variance  cumulative  at  pre¬ 
sent  is  5.4%  or  $11,759K.  This  is  an  increase  over  prior 
month  cumulative  data.  Current  month  cost  variance  is  8.2% 
and  $788K.  This  cost  problem  is  primarily  located  in  the 
airframe  which  at  present  is  9.6%  (8.4  million)  over  cost 
and  is  related  to  the  difficulty  with  the  chemical  structure 
of  the  composite  materials  being  used.  The  problem  pri¬ 
marily  affects  the  vertical  stabilizer  and  trailing  portion 
of  the  wings.  The  problem  is  the  result  of  coating  forming 
on  these  surfaces  as  a  result  of  the  exhaust  transfer.  We 
are  at  present  exploring  2  primary  alternatives,  1)  alter¬ 
ing  the  exhaust  heat  or  2)  altering  the  chemical  structure 
of  the  composite  material.  We  expect  resolution  by  June  and 
do  not  expect  any  significant  impact  to  the  overall  program. 

Response  Rating  4 
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Appendix  D:  Response  Evaluation  Package  (Page  32  of  37) 


Given  the  limited  information  provided  and  general  vari¬ 
ance  analysis  guidelines,  each  of  the  following  responses: 


1 

2 

3 

4 

5 

6 
7 


Contai ns 
Contains 
val ue . 
Contains 
Contains 
Contai ns 
Contai ns 
Contains 
tion . 


no  variance  analysis  value, 
a  very  limited  amount  of  variance  analysis 

a  limited  amount  of  variance  analysis  value, 
some  variance  analysis  value, 
a  moderate  amount  of  variance  analysis  value, 
a  large  amount  of  variance  analysis  value, 
a  variance  analysis  of  all  available  informa- 


061 

Three  areas  are  of  major  concern  for  BAC: 

1.  Airframe  the  greatest  variance  (14,424)  with  final 
and  future  problems  being  unknown. 

2.  Propulsion  the  next  greatest  variance  (2,983)  will  be 
solved  when  the  airframe  is  solved  so  it  is  a  dependent 
variable. 

3.  Systems  &  Test  is  the  third  highest  variance  (2,670) 
and  is  assumed  to  be  able  to  be  passed  and  come  up  to  speed. 

The  greatest  impact  is  the  airframe  which  will  also 
?ffect  the  propulsion  therefore,  more  emphasis  needs  to  be 
placed  in  this  area.  Reprogramming  may  be  necessary  to  get 
it  on  track. 


Response  Rating _ 1 


062 

Corrective  action  is  underway  to  resolve  the  airframe 
chemical  structure  problem  of  the  composite  material.  This 
will  involve  possible  modifications  to  the  propulsion  which 
will  require  possible  participation  in  design  change.  The 
present  impact  on  program  performance  is  noted  above. 

Response  Rating  1 
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Appendix  D:  Response  EvaTuation  Package  (Page  33  of  37) 


Given  the  limited  information  provided  and  general  vari¬ 
ance  analysis  guidelines,  each  of  the  following  responses: 


1 

2 

3 

4 

5 

6 
7 


Contains 
Contains 
value . 
Contains 
Contains 
Contains 
Contai ns 
Contains 
tion . 


no  variance  analysis  value, 
a  very  limited  amount  of  variance  analysis 

a  limited  amount  of  variance  analysis  value, 
some  variance  analysis  value, 
a  moderate  amount  of  variance  analysis  value, 
a  large  amount  of  variance  analysis  value, 
a  variance  analysis  of  all  available  informa- 


063 

After  careful  review  of  the  CPR  it  is  evident  that  the 
drivers  of  unfavorable  cost  variances  are  in  the  area  of  the 
Air  Vehicle.  The  problem  lies  in  the  level  3  production 
area,  namely  the  airframe  and  the  propulsion  system.  Regar¬ 
ding  the  airframe,  the  problem  is  the  chemical  structure  of 
the  composite  material.  When  the  exhaust  from  the  engines 
interacts  with  the  composite  material,  a  thin  coating  of 
residue  is  formed  on  the  surface  affecting  the  flight  char¬ 
acteristics  of  the  lifting  surfaces.  The  propulsion  vari¬ 
ance  is  essentially  due  to  the  airframe  problem  and  will  be 
resolved  when  the  corrective  action  of  the  airframe  is  in 
place  by  June.  The  System  Test  and  Evaluation  shows  an 
unfavorable  variance  due  to  the  airframe  and  propulsion 
problems.  The  next  System  Test  is  in  July.  No  problems  are 
anticipated  due  to  proposed  corrective  action  if  both  sys¬ 
tems  are  to  be  in  place  by  June. 


Response  Rating _ 3. 


064 

We  continue  to  have  problems  with  the  chemical  compo¬ 
sition  of  the  lifting  surfaces  and  the  propulsion  exhaust 
heat  temperature.  We  will  continue  to  take  the  necessary 
actions  to  resolve  the  problems  from  both  points  of  view. 
Both  Managers  are  certain  that  the  problems  can,  and  will, 
be  resolved  in  time  to  have  a  successful  systems  test  in 
July.  I  personally  will  work  with  each  of  the  Managers 
involved  to  assure  that  the  July  deadline  is  met. 

Response  Rating _ Z 
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Appendix  D:  Response  Evaluation  Package  (Page  34  of  37) 


Given  the  limited  information  provided  and  general  vari¬ 
ance  analysis  guidelines,  each  of  the  following  responses: 


1 

2 

3 

4 

5 

6 
7 


Contains 

Contains 

value. 

Contains 

Contains 

Contains 

Contains 

Contains 

tion . 


no  variance  analysis  value, 
a  very  limited  amount  of  variance  analysis 

a  limited  amount  of  variance  analysis  value, 
some  variance  analysis  value, 
a  moderate  amount  of  variance  analysis  value, 
a  large  amount  of  variance  analysis  value, 
a  variance  analysis  of  all  available  informa- 


065 


Technical  problems  with  the  Airframe  and  Propulsion  con¬ 
tinue  to  retard  progress.  These  problems  have  been  narrowed 
and  technical  solutions  are  being  addressed.  Solutions  are 
expected  by  June  allowing  no  further  impact  on  the  program. 
We  now  estimate  exceeding  our  BAC  by  $18M  but  $11M  remains 
in  MR  to  cover  a  portion  of  the  excess  over  contract  target 
cost. 

Response  Rating  3 

066 

Design  of  EGT  now  appears  to  be  appropriate.  The  Propul¬ 
sion  EAC  reflects  the  redesign  effort  completed.  Other 
Propulsion  areas  may  require  modification  which  will  prob¬ 
ably  further  increase  the  Propulsion  EAC.  These  impacts 
should  be  defined  and  quantified  by  June  and  will  be  cor¬ 
respondingly  reflected  in  the  Propulsion  EAC  at  that  time 
period.  The  Propulsion  impact  should  have  little  or  no 
impact  to  the  Airframe  design  or  EAC.  It  is  anticipated 
that  testing  will  start  in  July  with  no  further  impact  to 
testing.  In  summary,  the  current  cost  variances  and  program 
EAC  variances  are  expected  to  increase  and  must  be  quanti¬ 
fied  in  the  June  month-end  CPR. 

Response  Rating  2 


067 


We  are  experiencing  a  technical  problem  with  the  chemical 
structure  of  the  composite  material  on  the  lifting  surfaces. 
This  problem  has  affected  the  Airframe  Propulsion  and  Sys¬ 
tem,  Test  and  Evaluation.  WBS  elements  representing  sig¬ 
nificant  portion  of  the  cost  variance  (12,029).  It  is 
anticipated  that  the  problem  will  be  resolved  by  June, 
however,  it  is  doubtful  that  the  variance  in  cost  can/will 
be  recovered. 

Response  Rating  2 
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Appendix  D:  Response  Evaluation  Package  (Page  35  of  37) 


Given  the  limited  information  provided  and  general  vari¬ 
ance  analysis  guidelines,  each  of  the  following  responses: 


1 

2 

3 

4 

5 

6 
7 


Contains 
Contains 
value. 
Contai ns 
Contains 
Contains 
Contai ns 
Contai ns 
tion . 


no  variance  analysis  value, 
a  very  limited  amount  of  variance  analysis 

a  limited  amount  of  variance  analysis  value, 
some  variance  analysis  value, 
a  moderate  amount  of  variance  analysis  value, 
a  large  amount  of  variance  analysis  value, 
a  variance  analysis  of  all  available  informa- 


068 

Problem:  Program  director  states  problern  specifics  are 
unknown  and  has  asked  for  his  managers  to  look  into  the 
problems. 

Impact:  Recovery  not  expected. 


Response  Rating _ 1 


069 

Cost  variance  has  exceeded  the  scheduled  variance  by 
as  a  result  of  WBS  element  2  (airframe)  due  to  material 
chemical  structure  problems.  Also  WBS  element  2  (System 
Test  and  Evaluation)  also  (Propulsion)  WBS  element  have 
contributed  to  the  overall  effort  of  the  CWBS  variance. 
Measures  to  correct  these  problems  are  being  undertaken  and 
should  be  resolved  by  June. 


Response  Rating _ 1 


070 

Cost  variance  results  from  a  design  problem  with  the 
interaction  of  exhaust  heat  and  the  composite  structure  of 
the  airframe.  The  redesign  of  composite  material  should 
solve  the  problem  by  June  allowing  the  Systems  Test  to  take 
place  in  July.  If  the  redesign  is  successful,  the  total 
impact  should  not  exceed  5*  of  the  baseline  (w/o  MR). 

If  the  design  is  unsuccessful,  the  program  is  likely  to 
suffer  schedule  and  cost  impacts  that  are  undeterminable  at 
this  time. 


Response  Rating _ i. 
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Appendix  D:  Response  Evaluation  Package  (Page  36  of  37) 


Given  the  limited  information  provided  and  general  vari¬ 
ance  analysis  guidelines,  each  of  the  following  responses: 


1 

2 

3 

4 

5 

6 
7 


Contains 

Contains 

value. 

Contains 

Contains 

Contains 

Contains 

Contains 

tion . 


no  variance  analysis  value, 
a  very  limited  amount  of  variance  analysis 

a  limited  amount  of  variance  analysis  value, 
some  variance  analysis  value, 
a  moderate  amount  of  variance  analysis  value, 
a  large  amount  of  variance  analysis  value, 
a  variance  analysis  of  all  available  informa- 


071 


We  are  continuing  to  have  problems  with  the  chemical 
composition  of  the  lifting  surfaces  and  the  exhaust  gas 
temperature.  Will  task  Mr.  Marshall  and  C.  Anderson  to 
rectify  this  problem  within  30  days.  The  contract  will  see 
a  significant  increase  in  the  EAC  for  the  May  CPR.  This 
will  be  due  to  either  changes  in  the  chemical  composition 
and/or  exhaust  gas  temperature.  System  Test  and  Evaluation 
will  need  an  increase  in  budget  to  ensure  the  fix  is  ac¬ 
ceptable  and  pass  all  safety  requirements. 

Response  Rating  3 

072 

The  significant  problems  being  experienced  are  due  to  a 
chemical  reaction  between  the  engine  exhaust  and  the  lifting 
surface  composite  material.  The  Airframe  Manager  is  inves¬ 
tigating  the  alteration  of  the  chemical  structure  of  the 
composite  material  as  a  solution  to  the  problem  and  if 
successful,  will  resolve  the  problem  by  June  with  no  further 
impact  to  the  cost  variance.  The  Propulsion  Manager  is 
awaiting  the  results  of  the  investigation  prior  to  the 
modification  of  the  propulsion  system,  but  has  included 
potential  costs  of  the  investigation  in  the  current  EAC.  No 
additional  cost  degradation  is  expected. 

Response  Rating  4 

073 

Although  there  are  still  problems  in  the  Airframe  and 
Propulsion  WBSs,  these  should  be  satisfactorily  resolved  in 
June  with  minimum  impact  to  the  program.  The  additional 
costs  and  schedule  impacts  to  the  program  are  considered  to 
be  minimal . 

Response  Rating  1 
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Appendix  D:  Response  Evaluation  Package  (Page  37  of  37) 

Given  the  limited  information  provided  and  general  vari¬ 
ance  analysis  guidelines,  each  of  the  following  responses: 


1 

2 

3 

4 

5 

6 
7 


074 


Contains 
Contains 
value. 
Contains 
Contai ns 
Contains 
Contai ns 
Contains 
tion . 


no  variance  analysis  value, 
a  very  limited  amount  of  variance  analysis 

a  limited  amount  of  variance  analysis  value, 
some  variance  analysis  value, 
a  moderate  amount  of  variance  analysis  value, 
a  large  amount  of  variance  analysis  value, 
a  variance  analysis  of  all  available  informa- 


Problem:  Unanticipated  chemical  reaction  of  the  engine 

exhaust  with  the  composite  material  of  the  airframe  struc¬ 
ture.  This  causes  the  Airframe,  Propulsion,  and  Systems 
Test  and  Evaluation  variances. 


Impact:  These  reactions  reduce  the  air-worthiness  of 
the  aircraft  to  an  unacceptable  degree.  The  program  is 
behind  schedule  and  will  overrun  until  the  chemistry  is 
solved.  Management  reserve  will  be  distributed  for  the 
unanticipated  work  so  the  overall  contract  VAC  will  not 
change. 

Action  Taken:  The  Airframe,  Propulsion,  and  Systems 
Test  and  Evaluation  managers  are  cooperating  towards  a 
solution.  It  is  presently  anticipated  that  the  chemistry 
will  be  solved  in  June. 

Get  Well:  JHA  will  allocate  additional  resources  to 
solve  the  chemistry. 

Response  Rating _ g, 
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Appendix  F:  Tabulation  of  Response  Evaluations  By  Rater 


Instrument  Codes 

KF  =  Contractor  Feedback  GF  =  Government  Feedback 

KN  =  Contractor  Non-Feedback  GN  =  Government  Non-Feedback 
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Feedback  vs.  Non-Feedback  Test  Results 


Statistics : 


Total  Sample 

Score  Size  Mean  Variance 


Feedback  102 

Non-Feedback  106 


37  2.757 

37  2.865 


1 .745 
1  .065 


Hypothesis  Test  for  Equality  of  the  Sample  Means: 

Hjj;  -  Hy  =  0  There  is  no  statistical  difference 

between  feedback  and  non-feedback. 

H|;  py  -  py  ^  0  There  is  a  statistical  difference 

between  feedback  and  non-feedback. 


Decision  Rule: 

For  a  =  .10,  Reject  Hq  if: 


X  -  y _  — X  t-Ji 


[fi 

*  il 

or 

ZjI 

X 

c 

_J 

n,J 

n, 

m 

Results: 


^CALC  "  -.3918  and  ^TABlc  "  -1.6450, 

Therefore,  The  Null  Hypothesis  cannot  be  rejected  and 
there  is  no  statistical  difference  between  the  Feedback 
and  Non-Feedback  scores. 


127 


Bibliography 


1.  Abba,  Wayne.  "Cost/Schedule  Control  Systems  Criteria," 
Program  Manager.  15:  45-47  (November  -  December  1986). 

2.  Baumgartner,  Stanley  J.  "C/SCSC:  Alive  and  Well," 
Defense  Management  Journal.  10:  32-35  (April  1974). 

3.  Department  of  the  Air  Force.  Acquisition  Management. 
Program  Control  -  Financial.  AFR  800-6.  Washington: 
Government  Printing  Office,  13  September  1985. 

4.  Department  of  the  Air  Force.  Cost/Schedule  Control 
Systems  Criteria  Joint  Implementation  Guide.  AFSCP 
173-5.  Washington:  Government  Printing  Office,  1  October 
1987. 

5.  Department  of  Defense.  Cost  Performance  Report.  DOD 
Data  Item  Description  DI-F-6000C.  Washington: 

Government  Printing  Office,  1  December  1979. 

6.  Department  of  Defense.  Ma.lor  System  Acquisitions.  DOD 
Directive  5000.1.  Washington:  Government  Printing 
Office,  18  January  1977. 

7.  Department  of  Defense,  Performance  Measurement  for 
Selected  Acquisitions.  DOD  Directive  7000.2. 

Washington:  Government  Printing  Office,  10  June  1977. 

8.  Durso,  Francis  T.  and  Roger  L.  Mellgren.  Thinking  About 
Research .  St.  Paul,  MN :  West  Publishing  Co.  1989 

9.  Gadeken,  Owen  C.  and  Thomas  S.  Tison.  "The  Cost  of 
C/SCSC,"  Program  Manager.  12:  13-18  (July  -  August  1983). 

10.  Greller,  Martin  M.  "Evaluation  of  Feedback  Sources  as  a 
Function  of  Role  and  Organizational  Level,"  Journal  of 
Applied  Psychology.  65:  24-27  (February  1980) 

11.  Harrell,  Adrian  M.  "The  Decision-Making  Behavior  of  Air 
Force  Officers  and  the  Management  Control  Process," 
Accounting  Review.  52:  833-841  (October  1977). 

12.  Hemphill,  Ron  and  Mary  M.K.  Fleming.  "Cost/Schedule 

Management:  An  Earned  Value  Approach,"  Armed  Forces 

Comptroller.  32:  26-30  (Summer  1987). 

13.  Holeman,  Capt.  J.  B.  Jr.  "C/SCSC  Analysis:  The  Time  Is 
Now,"  Defense  Management  Journal .  1 :  39-42  (April  1974). 


128 


14.  Huczynski,  Andrzej.  Encyclopedia  of  Organizational 
Change  Methods.  Aldershot,  England:  Gower,  1987. 

15.  Komaki ,  Judi  and  others.  "Effect  of  Training  and 
Feedback:  Component  Analysis  of  a  Behavioral  Safety 
Program,"  Journal  of  Applied  Psychology.  65:  261-270 
(June  1980). 

16.  Marrella,  Lt.  Col.  Leonard  S.  The  Effect  of  the 
Cost/Schedule  Control  Systems  Criteria  on  Contractor 
Planning  and  Control.  PhD  dissertation.  The  George 
Washington  University,  Washington  DC,  February  1973 
(LD-30862  DLSIE). 

17.  Nadler,  David  A.  Feedback  and  Organization  Development: 
Using  Data-Based  Methods.  Reading  MA:  Addison-Wesley , 
1977 

18.  Roth,  Berton  J.  and  Capt  Wayne  C.  Hardin.  An  Analysis  of 
the  Development  Need  and  Theory  of  the  Air  Force 
Cost/Schedule  Planning  and  Control  Specif iciation.  MS 
thesis,  AFIT/LSR-1 0-68  School  of  Systems  and  Logistics, 
Air  Force  Institute  of  Technology  (AU),  Wri ght-Patterson 
AFB  OH,  August  1968  (AD-847730  DTIC). 

19.  Sul zer-Azaroff ,  Beth.  "Behavioral  Approaches  to 
Occupational  Health  and  Safety,"  Handbook  of 
Organizational  Behavior  Management,  edited  by  Lee  N. 
Frederiksen.  New  York:  Wiley  Interscience,  1982. 

20.  Tosi ,  Henry  L.  and  others.  Managing  Organization 
Behavior .  Marshfield,  MA:  Pitman,  1986. 

21.  Varady,  Joseph  R.  Jr.  and  Mark  J.  Lumer.  "Taking  the 
Heartburn  Out  of  C/SCSC,"  Program  Manager.  12:  11-12 
(July  -  August  1983). 

22.  Webster,  Dr.  Anthony.  "C/SCSC  Lessons  Learned 
Theoretical  Framework,"  Program  Manager.  17:  13-22  (July 
August  1988). 

23.  Weisberg,  Louis.  "C/SCSC:  Validation  Integrity 
Maintained  by  DCAS  Surveillance  Function,"  Defense 
Management  Journal.  10:  36-38  (April  1974). 

24.  Zemke,  Ronald  E.  and  John  M.  Gunkler.  "Organization  Wide 
Intervention,"  Handbook  of  Organizational  Behavior 
Management,  edited  by  Lee  N.  Frederiksen.  New  York: 

Wiley  Interscience,  1982. 


129 


Vita 


Captain  Juan  Amaral 


_ ....  ^  ,  -  -  ■  ,  -  was  accepted 

into  the  Air  Force  Academy  in  1980.  He  graduated  from 
the  Academy  in  1984  with  a  Bachelor  of  Science  in  Human 
Factors  Engineering.  Capt  Amaral  worked  as  a  program 
control  analyst  for  three  years  with  the  Defense  Support 
Program  at  Space  Division,  Los  Angeles  AFB,  prior  to  his 
assignment  at  AFIT  in  the  Cost  Analysis  program.  He  is 
married  to  the  former  Maria  Elena  Vera. 


130 


UNCLASSIFIED 


SECURITY  CL^SlFICATION  OF  THIS  PAGE 


la.  REPORT  SECURITY  CLASSIFICATION 

UNCLASSIFIED 


2a.  SECURITY  CLASSIFICATION  AUTHORITY 


2b.  DECLASSIFICATION /DOWNGRADING  SCHEDULE 


4.  PERFORMING  ORGANIZATION  REPORT  NUMBER(S) 

AFIT/GCA/LSY/89S-2 


REPORT  DOCUMENTATION  PAGE 


lb.  RESTRICTIVE  MARKINGS 


Form  Approved 
0MB  No.  0704-0183 


3.  DISTRIBUTION /AVAILABILITY  OF  REPORT 

Approved  for  public  release; 
distribution  unlimited 


S.  MONITORING  ORGANIZATION  REPORT  NUMBER(S) 


6a.  NAME  OF  PERFORMING  ORGANIZATION  6b.  OFFICE  SYMBOL  7a.  NAME  OF  MONITORING  ORGANIZATION 
School  of  Systems  (if appliceble) 

and  Logistics  AFIT/LSQ 


6c.  ADDRESS  {City,  State,  and  ZIP  Code) 

Air  Force  Institute  of  Technology  (AU 
V'Jr ight-Patterson  AFB  OH  45433-6583 


7b.  ADDRESS  (Ofy,  State,  and  ZIP  Code) 


Ba.  NAME  OF  FUNDING/SPONSORING 
ORGANIZATION 


8b.  OFFICE  SYMBOL  9  PROCUREMENT  INSTRUMENT  IDENTIFICATION  NUMBER 
(If  applicable) 


8c  ADDRESS  (C/ty,  State,  and  ZIP  Code) 


10.  SOURCE  OF  FUNDING  NUMBERS 


PROGRAM 
ELEMENT  NO. 


PROJECT 

TASK 

NO. 

NO 

.‘•13  Thesis 


'6.  supplementary  NOTaT'O*- 


COSATi  CODES 


G=0 


WORK  UNIT 
ACCESSION  NO. 


11.  TITLE  (Include  Security  Classification) 

THE  EFFECT  OF  FEEDBACK  ON  COST  PERFOR 

MANGE  REPORT  UTILITY 

•■2.  PERSONAL  AUTnO.R(S) 

Juan  H.  Amaral/  3.S.,  Captain/  US.AF 

‘8  SUBJECT  TERMS  {Continue  on  reverse  if  necessary  and  identify  by  block  number) 

Cost  Analysis  Contcaccing  Procurement 

Gost/Schecule  Control  Sysze.-.s  Criteria 


‘9  ABSTRACT  '.Continue  on  reverse  if  necessary  and  identify  by  block  number) 

Tnes  i 3  .Advisor:  .An t'no.ny  H  .  Presutt  i 

-nssistsr. t  rrotessor  or  .Accou; 


App^ved  for  pi^lic  ralease:  lAW  AFS  190-1. 

CARRY  /.  EKHELHAINZ,  Lt  Col,  USAF  11  Oct  89 
Director  of  Research  and  Consultation 
Air  Force  Institute  of  Technology  (AU) 
Wright-Patterson  AFB  OH  45433-6583 


20.  DISTRIBUTION /AVAILABILITY  OF  ABSTRACT  21.  ABSTRACT  SECURITY  CLASSIFICATION 

(3  UNCLASSIFIED/UNLIMITED  □  SAME  AS  RPT.  □  OTIC  USERS  UNCLASSIFIED 


22a.  NAME  OF  RESPONSIBLE  INDIVIDUAL  22b.  TELEPHONE  f/nc/ueie  Area  Code)  22c.  OFFICE  SYMBOL 

Anthony  H.  Presutti/  Asst  Professor  (513)255-6280  LSQ 


DD  Form  1473,  JUN  86  Previous  editions  are  obsolete.  SECURITY  CLASSIFICATION  OF  this  paG; 

UNCLASSIFIED 


UNCLASSIFIED 


The  purpose  of  this  thesis  was  to  assess  the  effect  of 
feedback  on  the  utility  of  the  Cost  Performance  Report 
(CPR).  The  effort  consisted  of  administering  a  controlled 
experiment  to  two  groups:  a  gathering  of  contractor  per¬ 
sonnel  and  a  group  of  Government  employees.  Both  groups  had 
some  degree  of  knowledge  in  the  area  of  variance  analysis 
reporting . 

Each  experiment  consisted  of  two  randomly  assigned 
instruments,  the  feedback  instrument  (given  to  the  experi¬ 
mental  group)  and  the  non-feedback  instrument  (given  to  the 
control  group). 

The  feedback  used  in  this  experiment  was  similar  in 
structure  to  the  Defense  Acquisition  Executive  Summary  For¬ 
mat  11,  specifically  as  formatted  in  the  Program  Director's 
Assessment  Review.  Both  of  these  reports  are  used  by 
government  program  offices  to  report  contract  cost  and 
schedule  information  to  higher  levels  of  the  government 
procurement  community. 

The  experiment  required  the  subjects  to  perform  a  task 
similar  to  that  performed  during  the  preparation  of  a  CPR 
and  to  provide  a  response  in  the  form  of  a  variance  analy¬ 
sis.  The  responses  were  rated  by  three  experts  and  the  mean 
values  of  the  feedback  responses  and  non-feedback  responses 
were  tested  to  determine  if  a  statistically  significant  dif¬ 
ference  existed  between  the  two  means. 

No  statistically  significant  difference  was  found  to 
exist  between  the  two  means  suggesting  that  providing  feed¬ 
back  to  subjects,  as  given  in  this  experiment,  has  no  effect 
on  the  quality  of  the  variance  analyses  they  prepare. 

Although  supporting  the  fact  that  feedback  as  a  whole 
is  beneficial,  the  results  of  this  thesis  suggest  that  in 
order  to  produce  a  statistically  significant  difference  in 
the  quality  of  variance  analyses,  the  feedback  given  in 
these  experiments  needs  to  be  reevaluated. 

A  recomimendation  was  made  to  conduct  a  follow-on 
thesis  with  an  improved  feedback  measure  possibly  involving 
a  feedback  briefing,  an  increase  in  time  allotted  to  the 
experiment,  or  a  change  to  the  feedback  vocabulary. 


UNCLASSIFIED 


